pull changes from dev to production on 2014-05-27

[cgolubi1]

This update brings these pulls, already on dev, to production:

• 689 Add color to overview page #836, Color and Column Adjustment #871: color on overview page
• Restricted mood values to standard die sizes #838, Added mad swing skill #839: mood swing dice only use standard die sizes, and add mad swing skill (doesn't appear in any available buttons yet)
• 800 editing htmllike chat #843: fix a bug in editing of chat containing html-like tags
• 193 display captured die #842: visually display captured dice
• Preferences to turn off button images and make the battlemat compact #851: add per-browser "compact" and "no images" settings
• 702 Visually distinguish the dev site from prod #849, apply dev site customization of UI look to dev.buttonweavers.com #873: visually distinguish the dev and prod sites
• Hiding game creation form after game creation #852: hide game creation form after game creation
• Account creation validation and form-hiding #853: do more validation of new usernames, and hide account creation form after account creation
• Highlight and resized chat #869, User activity timestamps #886, Base chat highlighting on active activity #887: highlight recent action log and chat messages
• Fixed QUnit tests #878: fix a couple of brittle QUnit tests
• Load all javascript dynamically #864: load all javascript files dynamically

Database updates:

00509_mad_swing_01.sql
00722_player_activity_01.sql

[cgolubi1]

Applying database changes: after "vagrant provision":

mysqldump -u root buttonmen > ~/buttonmen_db.$(timefmt).pre.sql
cd /buttonmen/deploy/database/updates
cat 00509_mad_swing_01.sql | mysql -u root buttonmen
cat 00722_player_activity_01.sql | mysql -u root buttonmen
mysqldump -u root buttonmen > ~/buttonmen_db.$(timefmt).post.sql

This tested out fine under vagrant.

[cgolubi1]

This wasn't good enough because of changes to the views which were not reflected in updates. After i updated using:

mysqldump -u root buttonmen > ~/buttonmen_db.$(timefmt).pre.sql
cd /buttonmen/deploy/database/updates
cat 00509_mad_swing_01.sql | mysql -u root buttonmen
cat 00722_player_activity_01.sql | mysql -u root buttonmen
mysqldump -u root buttonmen > ~/buttonmen_db.$(timefmt).post.sql

i got a PDO error loading any game because of:

Caught exception in BMInterface::load_game: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'v.last_action_time' in 'field list'

I shut down apache to try to fix it, and of course made things worse because i was working without a script in the database directory, not the updates directory, so i accidentally clobbered the "game" tables entirely:

sudo service apache2 stop
cd /buttonmen/deploy/database/updates
cat schema.game.sql | mysql -u root buttonmen

I then restored everything from the database backup i took after applying the previous updates, which should not have lost any data, recreated all the views (they should all be disposable, and "during an outage" is a bad time to go check which two i actually need), and restarted apache:

cat views.button.sql | mysql -u root buttonmen
cat views.config.sql | mysql -u root buttonmen
cat views.game.sql | mysql -u root buttonmen
cat views.player.sql | mysql -u root buttonmen

cd
mysqldump -u root buttonmen > buttonmen_db.`timefmt`.maybefixed.sql
sudo service apache2 start

Here's the diff between the "post" and maybefixed" versions:

www,[~],11:40(0)$ diff buttonmen_db.20140527.112528.post.sql buttonmen_db.20140527.113105.maybefixed.sql 
37449a37450
>   `last_action_time` tinyint NOT NULL,
38841a38843
>   `last_access_time` tinyint NOT NULL,
38906,38908c38908,38910
< /*!50001 SET character_set_client      = latin1 */;
< /*!50001 SET character_set_results     = latin1 */;
< /*!50001 SET collation_connection      = latin1_swedish_ci */;
---
> /*!50001 SET character_set_client      = utf8 */;
> /*!50001 SET character_set_results     = utf8 */;
> /*!50001 SET collation_connection      = utf8_general_ci */;
38911c38913
< /*!50001 VIEW `game_player_view` AS select `m`.`game_id` AS `game_id`,`m`.`player_id` AS `player_id`,`m`.`button_id` AS `button_id`,`m`.`alt_recipe` AS `alt_recipe`,`m`.`position` AS `position`,`m`.`did_win_initiative` AS `did_win_initiative`,`m`.`is_awaiting_action` AS `is_awaiting_action`,`m`.`n_rounds_won` AS `n_rounds_won`,`m`.`n_rounds_lost` AS `n_rounds_lost`,`m`.`n_rounds_drawn` AS `n_rounds_drawn`,`m`.`handicap` AS `handicap`,`m`.`is_player_hidden` AS `is_player_hidden`,`p`.`name_ingame` AS `player_name`,`p`.`autopass` AS `autopass`,`b`.`name` AS `button_name`,`g`.`n_target_wins` AS `n_target_wins` from (((`game_player_map` `m` left join `player` `p` on((`m`.`player_id` = `p`.`id`))) left join `button` `b` on((`m`.`button_id` = `b`.`id`))) left join `game` `g` on((`m`.`game_id` = `g`.`id`))) */;
---
> /*!50001 VIEW `game_player_view` AS select `m`.`game_id` AS `game_id`,`m`.`player_id` AS `player_id`,`m`.`button_id` AS `button_id`,`m`.`alt_recipe` AS `alt_recipe`,`m`.`position` AS `position`,`m`.`did_win_initiative` AS `did_win_initiative`,`m`.`is_awaiting_action` AS `is_awaiting_action`,`m`.`n_rounds_won` AS `n_rounds_won`,`m`.`n_rounds_lost` AS `n_rounds_lost`,`m`.`n_rounds_drawn` AS `n_rounds_drawn`,`m`.`handicap` AS `handicap`,`m`.`is_player_hidden` AS `is_player_hidden`,`m`.`last_action_time` AS `last_action_time`,`p`.`name_ingame` AS `player_name`,`p`.`autopass` AS `autopass`,`b`.`name` AS `button_name`,`g`.`n_target_wins` AS `n_target_wins` from (((`game_player_map` `m` left join `player` `p` on((`m`.`player_id` = `p`.`id`))) left join `button` `b` on((`m`.`button_id` = `b`.`id`))) left join `game` `g` on((`m`.`game_id` = `g`.`id`))) */;
38925,38927c38927,38929
< /*!50001 SET character_set_client      = latin1 */;
< /*!50001 SET character_set_results     = latin1 */;
< /*!50001 SET collation_connection      = latin1_swedish_ci */;
---
> /*!50001 SET character_set_client      = utf8 */;
> /*!50001 SET character_set_results     = utf8 */;
> /*!50001 SET collation_connection      = utf8_general_ci */;
38944,38946c38946,38948
< /*!50001 SET character_set_client      = latin1 */;
< /*!50001 SET character_set_results     = latin1 */;
< /*!50001 SET collation_connection      = latin1_swedish_ci */;
---
> /*!50001 SET character_set_client      = utf8 */;
> /*!50001 SET character_set_results     = utf8 */;
> /*!50001 SET collation_connection      = utf8_general_ci */;
38949c38951
< /*!50001 VIEW `player_view` AS select `a`.`auth_key` AS `auth_key`,`i`.`id` AS `id`,`i`.`name_ingame` AS `name_ingame`,`i`.`password_hashed` AS `password_hashed`,`i`.`name_irl` AS `name_irl`,`i`.`email` AS `email`,`i`.`status` AS `status`,`i`.`dob` AS `dob`,`i`.`autopass` AS `autopass`,`i`.`image_path` AS `image_path`,`i`.`comment` AS `comment`,`i`.`last_action_time` AS `last_action_time`,`i`.`creation_time` AS `creation_time`,`i`.`fanatic_button_id` AS `fanatic_button_id`,`i`.`n_games_won` AS `n_games_won`,`i`.`n_games_lost` AS `n_games_lost` from (`player` `i` left join `player_auth` `a` on((`i`.`id` = `a`.`id`))) */;
---
> /*!50001 VIEW `player_view` AS select `a`.`auth_key` AS `auth_key`,`i`.`id` AS `id`,`i`.`name_ingame` AS `name_ingame`,`i`.`password_hashed` AS `password_hashed`,`i`.`name_irl` AS `name_irl`,`i`.`email` AS `email`,`i`.`status` AS `status`,`i`.`dob` AS `dob`,`i`.`autopass` AS `autopass`,`i`.`image_path` AS `image_path`,`i`.`comment` AS `comment`,`i`.`last_action_time` AS `last_action_time`,`i`.`last_access_time` AS `last_access_time`,`i`.`creation_time` AS `creation_time`,`i`.`fanatic_button_id` AS `fanatic_button_id`,`i`.`n_games_won` AS `n_games_won`,`i`.`n_games_lost` AS `n_games_lost` from (`player` `i` left join `player_auth` `a` on((`i`.`id` = `a`.`id`))) */;
38963c38965
< -- Dump completed on 2014-05-27 11:25:30
---
> -- Dump completed on 2014-05-27 11:31:06

Are those "utf8" changes any cause for alarm? I think i'll wait and see if anyone complains.

According to the apache log, i'm the only who saw SQL errors while the database was up but in a bad state.

I don't think anyone could have advanced any games while the DB was up and in a bad state, so i don't think we lost any game data. I can also see from the backup that no player creation events happened to occur during that time. Could someone have created a game and lost it? Potentially, yes, but the apache logs show no activity from anyone except me after 11:24:24 UTC.

So i think we're safe, but i still see no activity from anyone except me in the logs, so i'm going to hang onto this until someone else advances a game.

[cgolubi1]

A second player successfully accessed the site, but i'm going to be nervous until i see more activity. Is it low right now because the new site is not very active yet in the early morning U.S. time, or is it low right now because people are seeing errors? Standing by.

[cgolubi1]

In terms of post-mortem, i'd say two things:

The important one is: the way we communicate about database updates which need to be pushed out during site updates is by putting files in deploy/database/updates/. We should treat this as the only valid way to communicate about database updates --- in this case, the original pull which changed the views contained text in the pull request saying to rerun the views. When i initially approved that pull and updated dev, i was looking at that pull request and its text. Today, i wasn't. Demonstrably, if there's no updates file, database updates get botched. So, yes, i think that means if you need to reload a view, you need to copy/paste the contents of the views file into a file under updates. This shouldn't lead to relevant bitrot, because updates only matter until they're applied.

So: as a pull submitter, if you're changing databases, make sure all changes are reflected in updates/ SQL files. As a pull approver, don't accept pulls if they have database changes which are missing from updates.

Secondarily: i didn't catch this, even though i tested the database changes by hand in vagrant, because my test suite wasn't rich enough --- i created some players and tested that the updates didn't blow up, but i didn't create any games or seriously traverse the UI. But that's always going to happen --- by-hand testing is a pain, and stuff will fall through the cracks. Odds are, whatever actually broke in any particular update will be whatever i happened not to test that time. So IMO we should stick with the first lesson --- the way we communicate about database updates which need to be pushed out during site updates is by putting files in deploy/database/updates/, period.

Any objections?