[WIP] re-enable solr-specific tests #1409

icarito · 2017-05-10T14:20:52Z

This is to replace #1386 with a branch in plots2 repository in order to enable coveralls.

all tests pass -- rake test:all
code is in uniquely-named feature branch, and has been rebased on top of latest master (especially if you've been asked to make additional changes)
pull request is descriptively named with #number reference back to original issue

icarito · 2017-05-10T14:21:43Z

PublicLabBot · 2017-05-10T14:30:55Z

	3 Messages
📖	@icarito Thank you for your pull request! I’m here to help with some tips and recommendations. Please take a look at the list provided and help us review and accept your contribution!
📖	This pull request doesn’t link to a issue number. Please refer to the issue it fixes (if any) in the body of your PR, in the format: `Fixes #123`.
📖	Pull Request is marked as Work in Progress

Generated by 🚫 Danger

icarito · 2017-05-10T14:42:26Z

Great so we can check the coveralls issue now. I'll be closing #1386 and follow this PR instead, @jywarren

icarito · 2017-05-10T16:35:54Z

Yay, figured out to fix the DISABLE_SOLR_CHECK (renamed) env var toggle to properly reindex in 47 minutes.

jywarren · 2017-05-10T16:55:07Z

wow, cool!

…

On Wed, May 10, 2017 at 12:35 PM, Sebastian Silva ***@***.***> wrote: Yay, figured out to fix the DISABLE_SOLR_CHECK (renamed) env var toggle to properly reindex in 47 minutes. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1409 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ1JzP9qHkEz3sZjbfqXm7W6OHIIrks5r4edqgaJpZM4NWvJ2> .

icarito · 2017-05-10T17:28:55Z

So it works:
http://branch1.laboratoriopublico.org/searches/55
Searches feel slow to me, possibly from our pre-search solr check.

icarito · 2017-05-10T18:50:06Z

This is the result of running solr tests in travis:

Command failed with status (1): [ruby -I"lib:test" -I"/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib" "/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/rake_test_loader.rb" "test/solr/search_record_test.rb" "test/solr/searches_controller_test.rb" ]
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/testtask.rb:109:in `block (3 levels) in define'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/file_utils.rb:57:in `call'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/file_utils.rb:57:in `sh'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/file_utils_ext.rb:37:in `sh'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/file_utils.rb:96:in `ruby'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/file_utils_ext.rb:37:in `ruby'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/testtask.rb:105:in `block (2 levels) in define'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/file_utils_ext.rb:58:in `verbose'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/testtask.rb:101:in `block in define'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:240:in `call'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:240:in `block in execute'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:235:in `each'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:235:in `execute'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
/usr/local/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:172:in `invoke_with_call_chain'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:165:in `invoke'
/usr/local/lib/ruby/gems/2.1.0/gems/railties-3.2.22.4/lib/rails/test_unit/testing.rake:50:in `block in <top (required)>'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:240:in `call'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:240:in `block in execute'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:235:in `each'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:235:in `execute'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
/usr/local/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:172:in `invoke_with_call_chain'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:165:in `invoke'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:150:in `invoke_task'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:106:in `block (2 levels) in top_level'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:106:in `each'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:106:in `block in top_level'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:115:in `run_with_threads'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:100:in `top_level'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:78:in `block in run'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:176:in `standard_exception_handling'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:75:in `run'
/usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/bin/rake:33:in `<top (required)>'
/usr/local/bin/rake:23:in `load'
/usr/local/bin/rake:23:in `<main>'
Tasks: TOP => test:single
[Coveralls] Using SimpleCov's default settings.
Loaded suite /usr/local/lib/ruby/gems/2.1.0/gems/rake-10.5.0/lib/rake/rake_test_loader
Started
....E
===============================================================================
Error: test_should_get_search_test_action(SearchesControllerTest): JSON::ParserError: 757: unexpected token at 'null'
test/solr/searches_controller_test.rb:23:in `block in <class:SearchesControllerTest>'
===============================================================================
....
Finished in 2.07917866 seconds.
------
9 tests, 17 assertions, 0 failures, 1 errors, 0 pendings, 0 omissions, 0 notifications
0% passed
------

jywarren · 2017-05-10T20:13:40Z

Huh, cool -- great work! But looks like the error at test/solr/searches_controller_test.rb:23 is distinct, maybe new? Shall we add that to the checklist too?

jywarren · 2017-05-10T20:14:47Z

But i got a bunch of email notifications about not working when Solr is off -- but the comments are gone now, was that resolved? Does it now work whether Solr is on or off?

icarito · 2017-05-10T20:51:57Z

Yes sorry - i deleted those messages - initially I accidentally double posted. Then I removed one copy. Then I looked at a different tab not knowing it and saw the duplicate message, so I deleted it, but turns out I deleted both messages. Sigh.

Summarizing, yes, stopping Solr causes an error. :-/

icarito · 2017-05-12T15:20:12Z

Here's the error we get when Solr is off:

Started GET "/searches/56" for 181.66.127.24 at 2017-05-12 15:17:55 +0000
Processing by SearchesController#show as HTML
  Parameters: {"id"=>"56"}
Completed 500 Internal Server Error in 67.8ms

RSolr::Error::ConnectionRefused (Connection refused - {:data=>"fq=type%3ANode&fq=updated_at_s%3A%7B*+TO+%222017%5C-05%5C-12%5C+15%5C%3A17%5C%3A55%5C+UTC%22%7D&q=balloon&fl=*+score&defType=edismax&start=0&rows=30&facet=true&f.updated_month_s.facet.mincount=1&facet.field=updated_month_s", :method=>:post, :params=>{:wt=>:ruby}, :query=>"wt=ruby", :headers=>{"Content-Type"=>"application/x-www-form-urlencoded; charset=UTF-8"}, :path=>"select", :uri=>#<URI::HTTP:0x007f6404245148 URL:http://pad.publiclab.org:8983/solr/default/select?wt=ruby>, :open_timeout=>nil, :read_timeout=>120, :retry_503=>nil, :retry_after_limit=>nil}):     app/models/search_record.rb:34:in `notes'                                                                                                                    app/controllers/searches_controller.rb:57:in `show'

icarito · 2017-05-12T16:35:31Z

Here's another coverage tool that perhaps even is used underneath coveralls?
https://github.com/colszowka/simplecov
It has pretty tables! ;-)

jywarren · 2017-05-12T17:53:58Z

Node.search is now being tested, could fail as of last couple commits.
we are trying to reindex embedded Solr in Travis rake test:solr, but that might not work?
it could need some sleep
maybe ditch Coveralls for simplecov

jywarren · 2017-05-12T18:03:10Z

Hmm, some green and red there, nice!

BTW does Coveralls just use SimpleCov? https://travis-ci.org/publiclab/plots2/builds/231645129#L2782 says:

[Coveralls] Using SimpleCov's default settings.

jywarren · 2017-05-12T18:05:52Z

missed an end -- fixed it and added one more commit!

jywarren · 2017-05-12T18:38:14Z

OK, reindex failed but because: sh: 1: docker-compose: not found on https://travis-ci.org/publiclab/plots2/builds/231657953#L2805

I think this is because we actually want to send it to embedded Solr, so we could just run rake SOLR_DISABLE_CHECK=1 sunspot:reindex without the docker compose stuff. Trying that.

jywarren · 2017-05-12T19:43:53Z

Nice! Solr indexed: [##################################] [72/72] [100.00%] [00:01] [00:00] [57.72/s]

Trying to resolve another test failure now...

jywarren · 2017-05-12T21:44:01Z

OK, in both cases it's not returning any results:


===============================================================================
Failure:
  <[]> expected to be != to
  <[]>.
test_should_get_search_test_action(SearchesControllerTest)
test/solr/searches_controller_test.rb:25:in `block in <class:SearchesControllerTest>'
===============================================================================
.F
===============================================================================
Failure: <false> is not true.
test_Node.search_for_two_different_key_words_returns_different_results(SearchRecordTest)
test/solr/search_record_test.rb:54:in `block in <class:SearchRecordTest>'
===============================================================================

However, Node.search does return results, so maybe we should just echo those out to be ...

OH! this is where we need htat extra parameter!

icarito · 2017-05-12T21:56:45Z

Yay, making progress! Good work!

…

On 12/05/17 16:44, Jeffrey Warren wrote: OK, in both cases it's not returning any results: |=============================================================================== Failure: <[]> expected to be != to <[]>. test_should_get_search_test_action(SearchesControllerTest) test/solr/searches_controller_test.rb:25:in `block in <class:SearchesControllerTest>' =============================================================================== .F =============================================================================== Failure: <false> is not true. test_Node.search_for_two_different_key_words_returns_different_results(SearchRecordTest) test/solr/search_record_test.rb:54:in `block in <class:SearchRecordTest>' =============================================================================== | However, |Node.search| does return results, so maybe we should just echo those out to be ... OH! this is where we need htat extra parameter! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1409 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMMS5UY9vEhnidO7HgeWuNTRU7p5wvBks5r5NKhgaJpZM4NWvJ2>.

jywarren · 2017-05-12T22:19:36Z

test/solr/search_record_test.rb

@@ -59,6 +59,8 @@ class SearchRecordTest < ActiveSupport::TestCase
      #with(:updated_month, month) if month.present?
      #paginate :page => 1, :per_page => 10
    end
+puts solr_search_1.results


Hmm, i have to wrap up for the day, but my attempt to output the results here didn't turn into anything. We could try puts solr_search_1.results.inspect instead...?

jywarren · 2017-05-13T23:47:01Z

Some hints on coveralls vs. simplecov here: http://technology.indiegogo.com/tag/simplecov/ -- though we should maybe just break out a separate issue.

jywarren · 2017-06-01T18:18:12Z

OK, after today's work, a checklist is:

we're outputting all the 72 indexed nodes from the Solr auto-testing, to see why "Chicago" and "pectro" are not returning any records in search_record_test.rb
@icarito is trying to get a rails console running with the test data so we can manually query it and see why it's not working in the tests.

We did confirm that in the staging server, these exact queries do return results, so the Solr test failures are confusing. All are related to Node.search do ... Solr queries.

jywarren · 2017-06-14T20:14:49Z

Rebased and attempted a fix

jywarren · 2017-06-14T20:40:58Z

aha -- hmm, this time, we dropped coverage again! how/why!?

jywarren · 2017-06-14T22:34:55Z

Awesome, this worked perfectly!

Solr search service offline

We can write a wrapper for Node.search which checks this first, such as Node.search_if_available!

jywarren · 2017-06-14T22:37:54Z

@icarito, if I add this, would you be ready to reenable Solr on production?

icarito · 2017-06-14T22:48:13Z

GREAT! pad.publiclab.org is ready with a Solr container.
I believe pushing to staging should work cleanly :-)
🏆 🌴 🎆 🥇 😀

icarito · 2017-06-14T22:51:07Z

Looking forward to put this in production. :-)

jywarren · 2017-06-14T23:28:02Z

Fingers X, i've implemented on-read toggles for all instances of Node.search.

jywarren · 2017-06-14T23:35:50Z

We could test this out on Sunday if you like -- assuming nothing else comes up. I want to be very thorough about testing with Solr on and off, in staging.

jywarren · 2017-06-14T23:35:56Z

And in production actually!

jywarren · 2017-06-14T23:38:27Z

Wheeee!

jywarren · 2017-06-14T23:39:59Z

Nice, and both http://branch1.laboratoriopublico.org/ and http://branch1.laboratoriopublico.org/searches/test/test?q=About load normally -- with Solr off. I think this is ready for a merge!

jywarren · 2017-06-15T00:09:00Z

docker-compose.yml

-      - docker-entrypoint.sh
-      - solr-precreate
-      - default
+#  solr:


Also, just checking -- do we need to re-enable this?

also any relation to #1428 ?

Not in production or staging as we're planning to deploy the solr container in pad.publiclab.org.

But I'll add a separate docker-compose-production.yml for this, so we might as well reenable this here.

jywarren · 2017-06-15T00:33:25Z

Do you want to make the required changes, as I'm not 100% sure I know -- and maybe add a comment accordingly inline? Then i'm ready to merge :-)

…

On Wed, Jun 14, 2017 at 8:30 PM, Sebastian Silva ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In docker-compose.yml <#1409 (comment)>: > @@ -13,16 +13,16 @@ services: - ../dump:/docker-entrypoint-initdb.d # for importing a dump # if you want to re-import simply put away ../mysql/* - solr: - image: solr:5.3 - ports: - - "127.0.0.1:8984:8983" - volumes: - - solr:/opt/solr/server/solr/mycores - entrypoint: - - docker-entrypoint.sh - - solr-precreate - - default +# solr: But I'll add a separate docker-compose-production.yml for this, so we might as well reenable this here. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1409 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ2gxHbUqYxJCrd0lg-rbA3n06D89ks5sEHtBgaJpZM4NWvJ2> .

icarito · 2017-06-15T00:36:21Z

It should work as-is, if I understood correctly. I'm going to be afk for the next few hours.
Regards,
Sebastian

jywarren · 2017-06-15T00:40:31Z

Ah, so i can merge this?

icarito · 2017-06-15T12:13:02Z

Yes, we can merge the commented version, I believe. Sunspot.yml doesn't point to the solr container but to localhost, and it's off by default as Solr is only optional. I'm adding an explanatory comment, though.

Just also add a note for the sake of thoroughness, the Solr schema creation process (which we assumed occurred during rake db:setup) showed differences when done with the embedded engine vs the container (to the point where queries required different parameters). It would be good to take these into consideration when putting in production, and even better, to understand why.

jywarren · 2017-06-15T12:15:56Z

Ok!!!

icarito · 2017-06-15T12:20:22Z

Oh wow that was fast I was committing the additional comment on docker-compose.yml -

jywarren added the in-progress label May 10, 2017

icarito mentioned this pull request May 10, 2017

re-enable solr-specific tests, next Solr steps #1386

Closed

10 tasks

icarito force-pushed the reenable-solr-tests branch 2 times, most recently from fa3e4b8 to e5baaf8 Compare May 10, 2017 15:18

icarito changed the title ~~re-enable solr-specific tests~~ [WIP] re-enable solr-specific tests May 10, 2017

icarito force-pushed the reenable-solr-tests branch from af44c53 to 5994757 Compare May 10, 2017 18:55

jywarren force-pushed the reenable-solr-tests branch from adbd92f to 9053929 Compare May 12, 2017 16:47

jywarren reviewed May 12, 2017

View reviewed changes

jywarren mentioned this pull request May 14, 2017

Add more functional tests to each method in Search API #1418

Closed

5 tasks

jywarren added 8 commits June 14, 2017 16:11

specifying solr fields

c87211d

test tweak

cedaba0

remove odd extra params tweak

da64335

more test tweaks

104bf6f

fixture tweak

d167be7

Create searches_controller.rb

f0bf0d0

Update searches_controller.rb

77770b0

attempted rejigger

fe76c6f

jywarren force-pushed the reenable-solr-tests branch from 602b473 to fe76c6f Compare June 14, 2017 20:14

adding on-read solr toggle

a213883

jywarren reviewed Jun 15, 2017

View reviewed changes

jywarren merged commit 6de3480 into master Jun 15, 2017

grvsachdeva mentioned this pull request Jan 22, 2018

Add more functional tests for each method in Search API #2072

Closed

[WIP] re-enable solr-specific tests #1409

[WIP] re-enable solr-specific tests #1409

Conversation

icarito commented May 10, 2017 • edited by jywarren Loading

icarito commented May 10, 2017 • edited by jywarren Loading

PublicLabBot commented May 10, 2017 • edited Loading

icarito commented May 10, 2017 • edited Loading

icarito commented May 10, 2017

jywarren commented May 10, 2017 via email

icarito commented May 10, 2017 • edited Loading

icarito commented May 10, 2017

jywarren commented May 10, 2017

jywarren commented May 10, 2017

icarito commented May 10, 2017

icarito commented May 12, 2017

icarito commented May 12, 2017

jywarren commented May 12, 2017

jywarren commented May 12, 2017

jywarren commented May 12, 2017

jywarren commented May 12, 2017

jywarren commented May 12, 2017

jywarren commented May 12, 2017

icarito commented May 12, 2017 via email

jywarren May 12, 2017

Choose a reason for hiding this comment

jywarren commented May 13, 2017

jywarren commented Jun 1, 2017

jywarren commented Jun 14, 2017

jywarren commented Jun 14, 2017

jywarren commented Jun 14, 2017

jywarren commented Jun 14, 2017

icarito commented Jun 14, 2017

icarito commented Jun 14, 2017

jywarren commented Jun 14, 2017

jywarren commented Jun 14, 2017

jywarren commented Jun 14, 2017

jywarren commented Jun 14, 2017

jywarren commented Jun 14, 2017

jywarren Jun 15, 2017

Choose a reason for hiding this comment

jywarren Jun 15, 2017

Choose a reason for hiding this comment

icarito Jun 15, 2017

Choose a reason for hiding this comment

icarito Jun 15, 2017

Choose a reason for hiding this comment

jywarren commented Jun 15, 2017 via email

icarito commented Jun 15, 2017

jywarren commented Jun 15, 2017

icarito commented Jun 15, 2017

jywarren commented Jun 15, 2017

icarito commented Jun 15, 2017

icarito commented May 10, 2017 •

edited by jywarren

Loading

icarito commented May 10, 2017 •

edited by jywarren

Loading

PublicLabBot commented May 10, 2017 •

edited

Loading

icarito commented May 10, 2017 •

edited

Loading

icarito commented May 10, 2017 •

edited

Loading