django-haystack-and-es

tech2024-04-02 69

https://github.com/Einsteinish/Django-Haystack-Elasticsearch/tree/master/search_app

https://zhuanlan.zhihu.com/p/34094103

elasticsearch模块: http://www.openskill.cn/article/47

elasticsearch是一个CS架构的软件,是一个分布式可扩展的实时搜索和分析引擎,也是一个框架除了后端在服务器上面装了elasticsearch搜索引擎之后, 还需要客户端使用elasticsearch-client与客户端进行交互

实现思路是: 首先要有一个MySQL数据库, 然后是ES服务器建立索引库

在后台启动es

[May@zanhu2 elasticsearch-7.9.0]$ touch running.log [May@zanhu2 elasticsearch-7.9.0]$ ./bin/elasticsearch > running.log 2>&1 & [1] 15508 [May@zanhu2 elasticsearch-7.9.0]$ ps -ef | grep java May 15508 14580 42 11:15 pts/0 00:00:25 /usr/java/jdk-14.0.2/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms512m -Xmx512m -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/elasticsearch-10984539820480802946 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -XX:MaxDirectMemorySize=268435456 -Des.path.home=/tmp/elasticsearch-7.9.0 -Des.path.conf=/tmp/elasticsearch-7.9.0/config -Des.distribution.flavor=default -Des.distribution.type=tar -Des.bundled_jdk=true -cp /tmp/elasticsearch-7.9.0/lib/* org.elasticsearch.bootstrap.Elasticsearch May 15759 14580 0 11:16 pts/0 00:00:00 grep --color=auto java [May@zanhu2 elasticsearch-7.9.0]$ exit exit (py37) [root@zanhu2 allauthdemo]# (py37) [root@zanhu2 allauthdemo]# (py37) [root@zanhu2 allauthdemo]# exit

note_text.txt 是需要索引的结果

free -m

当用户发表了一篇文章我们不可能随时在服务器上去跑一遍这个python manage.py rebuild_index这个命令,让它重新生成索引,我们必须要配置当数据库里面有新的数据增加或者修改或者是删除时,它能够实时去更新es索引库

,所以需要用到haystack里面的信号量机制

安装es踩过的坑：

https://www.cnblogs.com/ghw0501/p/12940039.html

自动killed: https://blog.csdn.net/chunmiao3032/article/details/107427407

https://blog.csdn.net/weixin_43093501/article/details/89920675

报错如下：

Unrecognized VM option 'UseParNewGC error：Could not create the Java Virtual Machine. Error:A Fatal exception has occurred,程序将退出

可能就是因为java的版本太高了,再安装一个合适的java,指定java环境变量就可以,就是说可以安装多个版本的java!

[May@zanhu2 bin]$ ./elasticsearch [2020-08-24T17:57:46,879][INFO ][o.e.n.Node ] [zanhu2] version[7.9.0], pid[506605], build[default/tar/a479a2a7fce0389512d6a9361301708b92dff667/2020-08-11T21:36:48.204330Z], OS[Linux/3.10.0-1127.13.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/14.0.2/14.0.2+12-46] [2020-08-24T17:57:46,908][INFO ][o.e.n.Node ] [zanhu2] JVM home [/usr/java/jdk-14.0.2] [2020-08-24T17:57:46,909][INFO ][o.e.n.Node ] [zanhu2] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-13774167505103428687, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/tmp/elasticsearch-7.9.0, -Des.path.conf=/tmp/elasticsearch-7.9.0/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true] [2020-08-24T17:57:53,310][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [aggs-matrix-stats] [2020-08-24T17:57:53,311][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [analysis-common] [2020-08-24T17:57:53,311][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [constant-keyword] [2020-08-24T17:57:53,311][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [flattened] [2020-08-24T17:57:53,311][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [frozen-indices] [2020-08-24T17:57:53,312][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [ingest-common] [2020-08-24T17:57:53,312][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [ingest-geoip] [2020-08-24T17:57:53,312][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [ingest-user-agent] [2020-08-24T17:57:53,312][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [kibana] [2020-08-24T17:57:53,315][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [lang-expression] [2020-08-24T17:57:53,315][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [lang-mustache] [2020-08-24T17:57:53,315][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [lang-painless] [2020-08-24T17:57:53,315][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [mapper-extras] [2020-08-24T17:57:53,316][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [parent-join] [2020-08-24T17:57:53,316][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [percolator] [2020-08-24T17:57:53,316][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [rank-eval] [2020-08-24T17:57:53,316][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [reindex] [2020-08-24T17:57:53,316][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [repository-url] [2020-08-24T17:57:53,317][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [search-business-rules] [2020-08-24T17:57:53,317][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [searchable-snapshots] [2020-08-24T17:57:53,317][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [spatial] [2020-08-24T17:57:53,317][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [tasks] [2020-08-24T17:57:53,322][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [transform] [2020-08-24T17:57:53,322][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [transport-netty4] [2020-08-24T17:57:53,323][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [vectors] [2020-08-24T17:57:53,323][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [wildcard] [2020-08-24T17:57:53,323][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-analytics] [2020-08-24T17:57:53,323][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-async] [2020-08-24T17:57:53,323][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-async-search] [2020-08-24T17:57:53,324][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-autoscaling] [2020-08-24T17:57:53,324][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-ccr] [2020-08-24T17:57:53,324][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-core] [2020-08-24T17:57:53,324][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-data-streams] [2020-08-24T17:57:53,324][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-deprecation] [2020-08-24T17:57:53,324][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-enrich] [2020-08-24T17:57:53,325][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-eql] [2020-08-24T17:57:53,325][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-graph] [2020-08-24T17:57:53,325][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-identity-provider] [2020-08-24T17:57:53,327][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-ilm] [2020-08-24T17:57:53,327][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-logstash] [2020-08-24T17:57:53,327][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-ml] [2020-08-24T17:57:53,327][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-monitoring] [2020-08-24T17:57:53,327][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-ql] [2020-08-24T17:57:53,328][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-rollup] [2020-08-24T17:57:53,328][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-security] [2020-08-24T17:57:53,328][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-sql] [2020-08-24T17:57:53,328][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-stack] [2020-08-24T17:57:53,328][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-voting-only-node] [2020-08-24T17:57:53,328][INFO ][o.e.p.PluginsService ] [zanhu2] loaded module [x-pack-watcher] [2020-08-24T17:57:53,331][INFO ][o.e.p.PluginsService ] [zanhu2] no plugins loaded [2020-08-24T17:57:53,522][INFO ][o.e.e.NodeEnvironment ] [zanhu2] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [21.4gb], net total_space [39.2gb], types [rootfs] [2020-08-24T17:57:53,523][INFO ][o.e.e.NodeEnvironment ] [zanhu2] heap size [1gb], compressed ordinary object pointers [true] [2020-08-24T17:57:53,740][INFO ][o.e.n.Node ] [zanhu2] node name [zanhu2], node ID [4mDodbjUSJmimuA9XVAuhw], cluster name [elasticsearch] Killed

安装es之前要先安装jdk

下载网址:http://apache.apooloo.cn/#/down/f1b54d77df533af9d9de55cf3f4e2cea

https://blog.csdn.net/baidu_20457783/article/details/89604867

安装步骤: https://www.php.cn/centos/445507.html

https://blog.csdn.net/u010590120/article/details/94736800

vim /etc/profile # 配置java的环境变量 JAVA_HOME=/usr/java/jdk1.8.0_202 CLASSPATH=$JAVA_HOME/lib/ PATH=$PATH:$JAVA_HOME/bin export PATH JAVA_HOME CLASSPATH source /etc/profile java -version

/tmp/elasticsearch-7.9.0/config/elasticsearch.yml

阿里云开放Elasticsearch:9200端口还是无法访问的解决办法

https://blog.csdn.net/yang958554999/article/details/105334505

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

elasticsearch启动时遇到的错误

问题翻译过来就是：elasticsearch用户拥有的内存权限太小，至少需要262144；

解决：切换到root用户执行命令： sysctl -w vm.max_map_count=262144 查看结果： sysctl -a|grep vm.max_map_count 显示： vm.max_map_count = 262144 上述方法修改之后，如果重启虚拟机将失效，所以：解决办法：在 /etc/sysctl.conf文件最后添加一行 vm.max_map_count=262144 即可永久修改

the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

https://blog.csdn.net/qq_43655835/article/details/104637625

通过客户端访问阿里云Elasticsearch

https://help.aliyun.com/document_detail/69194.html

from elasticsearch import Elasticsearch, RequestsHttpConnection import certifi es = Elasticsearch( ['<HOST>'], http_auth=('username', 'password'), port=9200, use_ssl=False ) res = es.index(index="my_index", doc_type="my_type", id=1, body={"title": "One", "tags": ["ruby"]}) res = es.get(index="my_index", doc_type="my_type", id=1) print(res['_source'])

django-contrib-comments

https://blog.csdn.net/weixin_30670151/article/details/96889561

https://github.com/xiaosongshine/djangoWebs

最新回复(0)