Data Update: Real-Time Data

Published on Author Scott Kennedy1 Comment

As you know I download the State Board’s data files every night. In most cases my data is current as of 9pm the night before. Usually the way this works is that the State Board makes a copy of their data available every night at 9pm and puts it on their FTP server and sometime around 2am my server copies those files and runs an update that lasts about 30-60 minutes and that’s how I display my data.

However the SBE also has a new filings list and makes that available in XML. I’ve been pulling each of those new filings and testing that download for a while now. For example this is the same page as their new filings page but with totals.

I’ve set it up so that my server now updates all of this data in real time. When it reads an A-1 or a B-1 it updates the data files with that info. So for example, last night Dale Fowler filed this A-1 at 10:59pm which is well past the 9pm cutoff (usually) for my data. Normally it would take a day before that showed up in my files (actually 2 because the SBE update doesn’t run on Saturday nights). Instead now it shows up right away in his financial profile and the race totals are also up to date. The only thing I can’t update in real time are D-2’s, in theory I could update the totals but I can’t get the itemized contributions and expenditures until I get those data files so it’s not worth pinging the SBE server so many times for so little benefit.

I tested this for a week before I put it live so I’m hopeful that this is working correctly but we’re coming up on the end of the quarter and the election isn’t too far away so it’s important to get things right. Please, please, please, if you see any errors or mistakes please let me know right away so I can correct them. There’s no worse feeling than getting facts wrong, I’m not that bright and I need all the help I can get.

Thank you

 

One Response to Data Update: Real-Time Data

  1. IRS SWAT teams? What for? What’s coming down the pike?Whats the problem? You like the idea of people carrying guns!More seriously, I fail to see what the issue is here. The IRS has a criminal enforcement division, law enforcement officals often carry guns. Certainly you’re not so silly as to believe the IRS never has to deal with criminal level tax evasion and going after drug dealers and organized crime doe&sn#8217;t require protection?

Leave a Reply

Your email address will not be published. Required fields are marked *