460213 - Deploy Sphinx to production

Status: RESOLVED INCOMPLETE

(support.mozilla.org :: Knowledge Base Software, task)

Description

[Nelson Ko [:nkoth]]

This is basically the first cut of the sumo search engine that should be fully functioning with memcache, but not optimized in any way yet. Must make 0.7.2 push.

Comment 1

[Nelson Ko [:nkoth]]

Invalid: do not really need all the initial changes in production to test (although parts will be in there, not everything will make it by 0.7.2 anyway).

(In reply to comment #0)
> This is basically the first cut of the sumo search engine that should be fully
> functioning with memcache, but not optimized in any way yet. Must make 0.7.2
> push.

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified, but I don't know why we couldn't have just left this open until we needed it.

Comment 3

[Laura Thomson :laura]

Leave it open, please.

Comment 4

[Nelson Ko [:nkoth]]

This bug is now tracking the deployment of Sphinx to production.

Comment 5

[Nelson Ko [:nkoth]]

Trevor,

Is Sphinx only on one webhead in production? https://bugzilla.mozilla.org/show_bug.cgi?id=464136#c13

Does this mean if I configure search to look for localhost port 3312, it won't work?

Do we want to do load testing (bug 460216) and then figure out how to configure a sphinx cluster (http://www.sphinxsearch.com/doc.html#distributed )? Or do we want to install Sphinx on all webheads?

Comment 6

[chizu]

It's on one webhead in production right now, so localhost will not work as is. There should be some load testing to decide if we need to partition or if simply deploying it on a few nodes for redundency can work. I think it's best to avoid deploying sphinx on all of them.

Comment 7

[Laura Thomson :laura]

Trevor, what kind of load testing did you have in mind?  We have a bug as per https://bugzilla.mozilla.org/show_bug.cgi?id=460216 but I don't think that's going to be specific enough for your needs.

Comment 8

[Laura Thomson :laura]

Nelson, who should this bug be assigned to?  Trevor?

Comment 9

[Nelson Ko [:nkoth]]

Trevor, if sphinxd is only on certain webheads, is there a load balancer address/port that we should be using?

Laura, the $host and $port that the search is using is currently in webroot/tiki-newsearch.php. Need a change in that to point to whatever is correct.

Also, perhaps we should move it out into scripts/sphinx/search.conf.php, which is included in tiki-newsearch.php?

Comment 10

[chizu]

The load balancer address and port should be moved to a config file. It's likely IT will need to adjust this from time to time, and a seperate local file prevents svn conflicts.

Comment 11

[Nelson Ko [:nkoth]]

I've created Bug 470365 for this and put it as 0.8.1. For now you can edit tiki-newsearch.php, right?

That way, we can figure out where the config file will be and all that in 0.8.1.


(In reply to comment #10)
> The load balancer address and port should be moved to a config file. It's
> likely IT will need to adjust this from time to time, and a seperate local file
> prevents svn conflicts.

Comment 12

[chizu]

Sphinx is deployed on production using round robin load balancing, but searchd isn't running.

# /usr/bin/php manual_index.php
Unknown column 'tiki_pages.keywords' in 'field list'

This errors out as in bug 468601:
# sudo -u apache /usr/bin/indexer --all
ERROR: index 'documents': source 'documents': XML parse error: no element found (line=2, pos=15, docid=0).

# sudo -u apache /usr/bin/searchd
WARNING: index 'documents': preload: failed to open /usr/local/sphinx/index/documents.sph: No such file or directory; NOT SERVING

Comment 13

[Nelson Ko [:nkoth]]

I think you might have forgotten to run https://bugzilla.mozilla.org/attachment.cgi?id=346962 on production.

Comment 14

[chizu]

Yep, missed that in my deploy notes from stage. That's now run.

Comment 15

[Nelson Ko [:nkoth]]

Problem in bug 470381 blocking this.

Comment 16

[Laura Thomson :laura]

Let's not run the indexer in prod again until we've tested it properly (Trevor, maybe you can help out with this on the test cluster?).  Dave had to truncate the se_words table this morning as it was bringing down the cluster.

Comment 17

[chizu]

I've disabled the indexer cron job in production.

Comment 18

[Nelson Ko [:nkoth]]

this refers to bug 470550

(In reply to comment #16)
> Let's not run the indexer in prod again until we've tested it properly (Trevor,
> maybe you can help out with this on the test cluster?).  Dave had to truncate
> the se_words table this morning as it was bringing down the cluster.

Comment 19

[Nelson Ko [:nkoth]]

Remember to configure scripts/sphinx/search.conf.php (not in svn) to point to searchd loadbalancer address/port

Comment 20

[Nelson Ko [:nkoth]]

Attachment: Instructions for first time search indexing

Follow these instructions to do a first time indexing. These were tested on the load test cluster.

Comment 21

[matthew zeier [:mrz]]

Little confused on what the next IT steps on this are.  Is this being wrapped up into another SUMO release?

Comment 22

[Laura Thomson :laura]

No, we'll do it between releases in a maintenance window.  The code is all in prod already, just not turned on yet.

Comment 23

[David Tenser [:djst]]

(In reply to comment #22)
> No, we'll do it between releases in a maintenance window.  The code is all in
> prod already, just not turned on yet.

Un-targeting from 1.0 per comment 22.

Comment 24

[matthew zeier [:mrz]]

Is there a schedule for this deployment?  Can this bug be closed and re-opened when ready?

Comment 25

[chizu]

Comment #20's instructions have been setup in production, but do not automatically run. Please reopen when we need to coordinate enabling this application side in production and having it run regularly.