The joys of data
I haven't written for a while now. That's because we actually got the data on the morning of day 67. The acceptance test of our software was on the afternoon of the day 68.
The data was severely broken, I complained, and in the evening of day 67 TomorrowDude sent me a better version.
I won't bore you with details. They keep fixing bits and pieces of data but after a month of their fixing it is still very broken, and lacking almost half of the records. The highlights include customers that exist in the data twice, uncharitable comments on customers in the username column, as opposed to the uncharitable-comment column, and that a certain one-to-one relation turns out to be one-to-many in many cases. After a month of nagging about fixing the latter problem they finally realized yesterday that:
a) X has many of Y in the data that they are sending us because X has many of Y in their database, even though it shouldn't,
b) they really should fix their database, because every X should have only one Y and vice versa,
c) nobody has any clue as to how the extra Y got into the database,
d) they have to remove all the extra Y, but nobody seems to have any clue as to which ones are supposed to be there and which ones are extra,
e) since the data with more than one Y per X is frowned upon by the XML parser, they have quietly modified the .xsd file to include more than one Y per X, without saying a word to us.
Kukkahattutäti asked me for the nth time to send her the records that are broken, and group them nicely, if I would. I did it for the nth time (just the grep results, nothing fancy), but pointed out that this was not my job, but hers. I think I am becoming rude.
Koodiorjatyttö: I think that at this point you should hire somebody to check your data integrity.
Kukkahattutäti: We have 40 people checking the data integrity now.
Koodiorjatyttö (aside): Yeah, and I have a friend who is checking human rights in Chechnya, and much with the same result.
ProjectManager (aside to Koodiorjatyttö): Tell her to fire them all and hire one who can.
FriendOnIrc (to Koodiorjatyttö): Tell her to fire them all and hire one who can.
The data was severely broken, I complained, and in the evening of day 67 TomorrowDude sent me a better version.
I won't bore you with details. They keep fixing bits and pieces of data but after a month of their fixing it is still very broken, and lacking almost half of the records. The highlights include customers that exist in the data twice, uncharitable comments on customers in the username column, as opposed to the uncharitable-comment column, and that a certain one-to-one relation turns out to be one-to-many in many cases. After a month of nagging about fixing the latter problem they finally realized yesterday that:
a) X has many of Y in the data that they are sending us because X has many of Y in their database, even though it shouldn't,
b) they really should fix their database, because every X should have only one Y and vice versa,
c) nobody has any clue as to how the extra Y got into the database,
d) they have to remove all the extra Y, but nobody seems to have any clue as to which ones are supposed to be there and which ones are extra,
e) since the data with more than one Y per X is frowned upon by the XML parser, they have quietly modified the .xsd file to include more than one Y per X, without saying a word to us.
Kukkahattutäti asked me for the nth time to send her the records that are broken, and group them nicely, if I would. I did it for the nth time (just the grep results, nothing fancy), but pointed out that this was not my job, but hers. I think I am becoming rude.
Koodiorjatyttö: I think that at this point you should hire somebody to check your data integrity.
Kukkahattutäti: We have 40 people checking the data integrity now.
Koodiorjatyttö (aside): Yeah, and I have a friend who is checking human rights in Chechnya, and much with the same result.
ProjectManager (aside to Koodiorjatyttö): Tell her to fire them all and hire one who can.
FriendOnIrc (to Koodiorjatyttö): Tell her to fire them all and hire one who can.
