Storing the pre-calculated user and items vectors of movielens dataset into redis in-memory database and then indexing into milvus for efficient large-scale retrieval
Packages | Servers |
---|---|
pymilvus | milvus-1.1.0 |
redis | redis |
paddle_serving_app | |
paddlepaddle |
!pip install pymilvus==1.1.0
!pip install paddle_serving_app==0.3.1
!pip install paddlepaddle
!pip install redis
Warning: It will take ~40 minutes to install!
!git clone -b 1.1 https://github.com/milvus-io/milvus.git
% cd /content/milvus/core
! ./ubuntu_build_deps.sh
!./build.sh -t Release
# !./build.sh -t Release -g
% cd /content/milvus/core/milvus
! echo $LD_LIBRARY_PATH
import os
os.environ['LD_LIBRARY_PATH'] +=":/content/milvus/core/milvus/lib"
! echo $LD_LIBRARY_PATH
% cd scripts
! nohup ./start_server.sh &
! cat nohup.out
!cat nohup.out
__ _________ _ ____ ______ / |/ / _/ /| | / / / / / __/ / /|_/ // // /_| |/ / /_/ /\ \ /_/ /_/___/____/___/\____/___/ Welcome to use Milvus! Milvus Release version: v1.1.1, built at 2021-06-23 14:11.42, with OpenBLAS library. You are using Milvus CPU edition Last commit id: 3fc81236452d8060fe7adc1793ad1d69f3d8423c Loading configuration from: ../conf/server_config.yaml NOTICE: You are using SQLite as the meta data management. We recommend change it to MySQL. Supported CPU instruction sets: avx2, sse4_2 FAISS hook AVX2 Milvus server started successfully! Milvus server is going to shutdown ... Milvus server exit...
We are using Redis as a metadata storage service. Code can easily be modified to use a python dictionary, but that usually does not work in any use case outside of quick examples. We need a metadata storage service in order to be able to be able to map between embeddings and the corresponding data.
#hide-output
!wget http://download.redis.io/releases/redis-stable.tar.gz --no-check-certificate
!tar -xf redis-stable.tar.gz && cd redis-stable/src && make
--2021-06-23 14:52:56-- http://download.redis.io/releases/redis-stable.tar.gz Resolving download.redis.io (download.redis.io)... 45.60.121.1 Connecting to download.redis.io (download.redis.io)|45.60.121.1|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 2261060 (2.2M) [application/octet-stream] Saving to: ‘redis-stable.tar.gz’ redis-stable.tar.gz 100%[===================>] 2.16M --.-KB/s in 0.1s 2021-06-23 14:52:57 (21.0 MB/s) - ‘redis-stable.tar.gz’ saved [2261060/2261060] CC Makefile.dep rm -rf redis-server redis-sentinel redis-cli redis-benchmark redis-check-rdb redis-check-aof *.o *.gcda *.gcno *.gcov redis.info lcov-html Makefile.dep dict-benchmark rm -f adlist.d quicklist.d ae.d anet.d dict.d server.d sds.d zmalloc.d lzf_c.d lzf_d.d pqsort.d zipmap.d sha1.d ziplist.d release.d networking.d util.d object.d db.d replication.d rdb.d t_string.d t_list.d t_set.d t_zset.d t_hash.d config.d aof.d pubsub.d multi.d debug.d sort.d intset.d syncio.d cluster.d crc16.d endianconv.d slowlog.d scripting.d bio.d rio.d rand.d memtest.d crcspeed.d crc64.d bitops.d sentinel.d notify.d setproctitle.d blocked.d hyperloglog.d latency.d sparkline.d redis-check-rdb.d redis-check-aof.d geo.d lazyfree.d module.d evict.d expire.d geohash.d geohash_helper.d childinfo.d defrag.d siphash.d rax.d t_stream.d listpack.d localtime.d lolwut.d lolwut5.d lolwut6.d acl.d gopher.d tracking.d connection.d tls.d sha256.d timeout.d setcpuaffinity.d anet.d adlist.d dict.d redis-cli.d zmalloc.d release.d ae.d crcspeed.d crc64.d siphash.d crc16.d ae.d anet.d redis-benchmark.d adlist.d dict.d zmalloc.d siphash.d (cd ../deps && make distclean) make[1]: Entering directory '/content/milvus/core/milvus/scripts/redis-stable/deps' (cd hiredis && make clean) > /dev/null || true (cd linenoise && make clean) > /dev/null || true (cd lua && make clean) > /dev/null || true (cd jemalloc && [ -f Makefile ] && make distclean) > /dev/null || true (rm -f .make-*) make[1]: Leaving directory '/content/milvus/core/milvus/scripts/redis-stable/deps' (rm -f .make-*) echo STD=-std=c11 -pedantic -DREDIS_STATIC='' >> .make-settings echo WARN=-Wall -W -Wno-missing-field-initializers >> .make-settings echo OPT=-O2 >> .make-settings echo MALLOC=jemalloc >> .make-settings echo CFLAGS= >> .make-settings echo LDFLAGS= >> .make-settings echo REDIS_CFLAGS= >> .make-settings echo REDIS_LDFLAGS= >> .make-settings echo PREV_FINAL_CFLAGS=-std=c11 -pedantic -DREDIS_STATIC='' -Wall -W -Wno-missing-field-initializers -O2 -g -ggdb -I../deps/hiredis -I../deps/linenoise -I../deps/lua/src -DUSE_JEMALLOC -I../deps/jemalloc/include >> .make-settings echo PREV_FINAL_LDFLAGS= -g -ggdb -rdynamic >> .make-settings (cd ../deps && make hiredis linenoise lua jemalloc) make[1]: Entering directory '/content/milvus/core/milvus/scripts/redis-stable/deps' (cd hiredis && make clean) > /dev/null || true (cd linenoise && make clean) > /dev/null || true (cd lua && make clean) > /dev/null || true (cd jemalloc && [ -f Makefile ] && make distclean) > /dev/null || true (rm -f .make-*) (echo "" > .make-cflags) (echo "" > .make-ldflags) MAKE hiredis cd hiredis && make static make[2]: Entering directory '/content/milvus/core/milvus/scripts/redis-stable/deps/hiredis' cc -std=c99 -pedantic -c -O3 -fPIC -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb net.c cc -std=c99 -pedantic -c -O3 -fPIC -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb hiredis.c cc -std=c99 -pedantic -c -O3 -fPIC -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb sds.c cc -std=c99 -pedantic -c -O3 -fPIC -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb async.c cc -std=c99 -pedantic -c -O3 -fPIC -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb read.c cc -std=c99 -pedantic -c -O3 -fPIC -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb sockcompat.c ar rcs libhiredis.a net.o hiredis.o sds.o async.o read.o sockcompat.o make[2]: Leaving directory '/content/milvus/core/milvus/scripts/redis-stable/deps/hiredis' MAKE linenoise cd linenoise && make make[2]: Entering directory '/content/milvus/core/milvus/scripts/redis-stable/deps/linenoise' cc -Wall -Os -g -c linenoise.c make[2]: Leaving directory '/content/milvus/core/milvus/scripts/redis-stable/deps/linenoise' MAKE lua cd lua/src && make all CFLAGS="-O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' " MYLDFLAGS="" AR="ar rcu" make[2]: Entering directory '/content/milvus/core/milvus/scripts/redis-stable/deps/lua/src' cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lapi.o lapi.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lcode.o lcode.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o ldebug.o ldebug.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o ldo.o ldo.c ldo.c: In function ‘f_parser’: ldo.c:496:7: warning: unused variable ‘c’ [-Wunused-variable] int c = luaZ_lookahead(p->z); ^ cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o ldump.o ldump.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lfunc.o lfunc.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lgc.o lgc.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o llex.o llex.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lmem.o lmem.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lobject.o lobject.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lopcodes.o lopcodes.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lparser.o lparser.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lstate.o lstate.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lstring.o lstring.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o ltable.o ltable.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o ltm.o ltm.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lundump.o lundump.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lvm.o lvm.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lzio.o lzio.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o strbuf.o strbuf.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o fpconv.o fpconv.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lauxlib.o lauxlib.c lauxlib.c: In function ‘luaL_loadfile’: lauxlib.c:577:4: warning: this ‘while’ clause does not guard... [-Wmisleading-indentation] while ((c = getc(lf.f)) != EOF && c != LUA_SIGNATURE[0]) ; ^~~~~ lauxlib.c:578:5: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘while’ lf.extraline = 0; ^~ cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lbaselib.o lbaselib.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o ldblib.o ldblib.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o liolib.o liolib.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lmathlib.o lmathlib.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o loslib.o loslib.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o ltablib.o ltablib.c ltablib.c: In function ‘addfield’: ltablib.c:137:3: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation] if (!lua_isstring(L, -1)) ^~ ltablib.c:140:5: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’ luaL_addvalue(b); ^~~~~~~~~~~~~ cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lstrlib.o lstrlib.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o loadlib.o loadlib.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o linit.o linit.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lua_cjson.o lua_cjson.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lua_struct.o lua_struct.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lua_cmsgpack.o lua_cmsgpack.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lua_bit.o lua_bit.c ar rcu liblua.a lapi.o lcode.o ldebug.o ldo.o ldump.o lfunc.o lgc.o llex.o lmem.o lobject.o lopcodes.o lparser.o lstate.o lstring.o ltable.o ltm.o lundump.o lvm.o lzio.o strbuf.o fpconv.o lauxlib.o lbaselib.o ldblib.o liolib.o lmathlib.o loslib.o ltablib.o lstrlib.o loadlib.o linit.o lua_cjson.o lua_struct.o lua_cmsgpack.o lua_bit.o # DLL needs all object files ar: `u' modifier ignored since `D' is the default (see `U') ranlib liblua.a cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lua.o lua.c cc -o lua lua.o liblua.a -lm liblua.a(loslib.o): In function `os_tmpname': loslib.c:(.text+0x290): warning: the use of `tmpnam' is dangerous, better use `mkstemp' cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o luac.o luac.c cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o print.o print.c cc -o luac luac.o print.o liblua.a -lm make[2]: Leaving directory '/content/milvus/core/milvus/scripts/redis-stable/deps/lua/src' MAKE jemalloc cd jemalloc && ./configure --with-version=5.1.0-0-g0 --with-lg-quantum=3 --with-jemalloc-prefix=je_ --enable-cc-silence CFLAGS="-std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops " LDFLAGS="" configure: WARNING: unrecognized options: --enable-cc-silence checking for xsltproc... false checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking whether compiler is cray... no checking whether compiler supports -std=gnu11... yes checking whether compiler supports -Wall... yes checking whether compiler supports -Wshorten-64-to-32... no checking whether compiler supports -Wsign-compare... yes checking whether compiler supports -Wundef... yes checking whether compiler supports -Wno-format-zero-length... yes checking whether compiler supports -pipe... yes checking whether compiler supports -g3... yes checking how to run the C preprocessor... gcc -E checking for g++... g++ checking whether we are using the GNU C++ compiler... yes checking whether g++ accepts -g... yes checking whether g++ supports C++14 features by default... yes checking whether compiler supports -Wall... yes checking whether compiler supports -g3... yes checking whether libstdc++ linkage is compilable... yes checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking whether byte ordering is bigendian... no checking size of void *... 8 checking size of int... 4 checking size of long... 8 checking size of long long... 8 checking size of intmax_t... 8 checking build system type... x86_64-pc-linux-gnu checking host system type... x86_64-pc-linux-gnu checking whether pause instruction is compilable... yes checking number of significant virtual address bits... 48 checking for ar... ar checking for nm... nm checking for gawk... no checking for mawk... mawk checking malloc.h usability... yes checking malloc.h presence... yes checking for malloc.h... yes checking whether malloc_usable_size definition can use const argument... no checking for library containing log... -lm checking whether __attribute__ syntax is compilable... yes checking whether compiler supports -fvisibility=hidden... yes checking whether compiler supports -fvisibility=hidden... yes checking whether compiler supports -Werror... yes checking whether compiler supports -herror_on_warning... no checking whether tls_model attribute is compilable... yes checking whether compiler supports -Werror... yes checking whether compiler supports -herror_on_warning... no checking whether alloc_size attribute is compilable... yes checking whether compiler supports -Werror... yes checking whether compiler supports -herror_on_warning... no checking whether format(gnu_printf, ...) attribute is compilable... yes checking whether compiler supports -Werror... yes checking whether compiler supports -herror_on_warning... no checking whether format(printf, ...) attribute is compilable... yes checking for a BSD-compatible install... /usr/bin/install -c checking for ranlib... ranlib checking for ld... /usr/bin/ld checking for autoconf... false checking for memalign... yes checking for valloc... yes checking whether compiler supports -O3... yes checking whether compiler supports -O3... yes checking whether compiler supports -funroll-loops... yes checking configured backtracing method... N/A checking for sbrk... yes checking whether utrace(2) is compilable... no checking whether a program using __builtin_unreachable is compilable... yes checking whether a program using __builtin_ffsl is compilable... yes checking LG_PAGE... 12 checking pthread.h usability... yes checking pthread.h presence... yes checking for pthread.h... yes checking for pthread_create in -lpthread... yes checking dlfcn.h usability... yes checking dlfcn.h presence... yes checking for dlfcn.h... yes checking for dlsym... no checking for dlsym in -ldl... yes checking whether pthread_atfork(3) is compilable... yes checking whether pthread_setname_np(3) is compilable... yes checking for library containing clock_gettime... none required checking whether clock_gettime(CLOCK_MONOTONIC_COARSE, ...) is compilable... yes checking whether clock_gettime(CLOCK_MONOTONIC, ...) is compilable... yes checking whether mach_absolute_time() is compilable... no checking whether compiler supports -Werror... yes checking whether syscall(2) is compilable... yes checking for secure_getenv... yes checking for sched_getcpu... yes checking for sched_setaffinity... yes checking for issetugid... no checking for _malloc_thread_cleanup... no checking for _pthread_mutex_init_calloc_cb... no checking for TLS... yes checking whether C11 atomics is compilable... no checking whether GCC __atomic atomics is compilable... yes checking whether GCC __sync atomics is compilable... yes checking whether Darwin OSAtomic*() is compilable... no checking whether madvise(2) is compilable... yes checking whether madvise(..., MADV_FREE) is compilable... yes checking whether madvise(..., MADV_DONTNEED) is compilable... yes checking whether madvise(..., MADV_DO[NT]DUMP) is compilable... yes checking whether madvise(..., MADV_[NO]HUGEPAGE) is compilable... yes checking whether to force 32-bit __sync_{add,sub}_and_fetch()... no checking whether to force 64-bit __sync_{add,sub}_and_fetch()... no checking for __builtin_clz... yes checking whether Darwin os_unfair_lock_*() is compilable... no checking whether Darwin OSSpin*() is compilable... no checking whether glibc malloc hook is compilable... yes checking whether glibc memalign hook is compilable... yes checking whether pthreads adaptive mutexes is compilable... yes checking whether compiler supports -D_GNU_SOURCE... yes checking whether compiler supports -Werror... yes checking whether compiler supports -herror_on_warning... no checking whether strerror_r returns char with gnu source is compilable... yes checking for stdbool.h that conforms to C99... yes checking for _Bool... yes configure: creating ./config.status config.status: creating Makefile config.status: creating jemalloc.pc config.status: creating doc/html.xsl config.status: creating doc/manpages.xsl config.status: creating doc/jemalloc.xml config.status: creating include/jemalloc/jemalloc_macros.h config.status: creating include/jemalloc/jemalloc_protos.h config.status: creating include/jemalloc/jemalloc_typedefs.h config.status: creating include/jemalloc/internal/jemalloc_preamble.h config.status: creating test/test.sh config.status: creating test/include/test/jemalloc_test.h config.status: creating config.stamp config.status: creating bin/jemalloc-config config.status: creating bin/jemalloc.sh config.status: creating bin/jeprof config.status: creating include/jemalloc/jemalloc_defs.h config.status: creating include/jemalloc/internal/jemalloc_internal_defs.h config.status: creating test/include/test/jemalloc_test_defs.h config.status: executing include/jemalloc/internal/public_symbols.txt commands config.status: executing include/jemalloc/internal/private_symbols.awk commands config.status: executing include/jemalloc/internal/private_symbols_jet.awk commands config.status: executing include/jemalloc/internal/public_namespace.h commands config.status: executing include/jemalloc/internal/public_unnamespace.h commands config.status: executing include/jemalloc/internal/size_classes.h commands config.status: executing include/jemalloc/jemalloc_protos_jet.h commands config.status: executing include/jemalloc/jemalloc_rename.h commands config.status: executing include/jemalloc/jemalloc_mangle.h commands config.status: executing include/jemalloc/jemalloc_mangle_jet.h commands config.status: executing include/jemalloc/jemalloc.h commands configure: WARNING: unrecognized options: --enable-cc-silence =============================================================================== jemalloc version : 5.1.0-0-g0 library revision : 2 CONFIG : --with-version=5.1.0-0-g0 --with-lg-quantum=3 --with-jemalloc-prefix=je_ --enable-cc-silence 'CFLAGS=-std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops ' LDFLAGS= CC : gcc CONFIGURE_CFLAGS : -std=gnu11 -Wall -Wsign-compare -Wundef -Wno-format-zero-length -pipe -g3 -fvisibility=hidden -O3 -funroll-loops SPECIFIED_CFLAGS : -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops EXTRA_CFLAGS : CPPFLAGS : -D_GNU_SOURCE -D_REENTRANT CXX : g++ CONFIGURE_CXXFLAGS : -Wall -g3 -fvisibility=hidden -O3 SPECIFIED_CXXFLAGS : EXTRA_CXXFLAGS : LDFLAGS : EXTRA_LDFLAGS : DSO_LDFLAGS : -shared -Wl,-soname,$(@F) LIBS : -lm -lstdc++ -lpthread -ldl RPATH_EXTRA : XSLTPROC : false XSLROOT : PREFIX : /usr/local BINDIR : /usr/local/bin DATADIR : /usr/local/share INCLUDEDIR : /usr/local/include LIBDIR : /usr/local/lib MANDIR : /usr/local/share/man srcroot : abs_srcroot : /content/milvus/core/milvus/scripts/redis-stable/deps/jemalloc/ objroot : abs_objroot : /content/milvus/core/milvus/scripts/redis-stable/deps/jemalloc/ JEMALLOC_PREFIX : je_ JEMALLOC_PRIVATE_NAMESPACE : je_ install_suffix : malloc_conf : autogen : 0 debug : 0 stats : 1 prof : 0 prof-libunwind : 0 prof-libgcc : 0 prof-gcc : 0 fill : 1 utrace : 0 xmalloc : 0 log : 0 lazy_lock : 0 cache-oblivious : 1 cxx : 1 =============================================================================== cd jemalloc && make CFLAGS="-std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops " LDFLAGS="" lib/libjemalloc.a make[2]: Entering directory '/content/milvus/core/milvus/scripts/redis-stable/deps/jemalloc' gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/jemalloc.sym.o src/jemalloc.c nm -a src/jemalloc.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/jemalloc.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/arena.sym.o src/arena.c nm -a src/arena.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/arena.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/background_thread.sym.o src/background_thread.c nm -a src/background_thread.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/background_thread.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/base.sym.o src/base.c nm -a src/base.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/base.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/bin.sym.o src/bin.c nm -a src/bin.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/bin.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/bitmap.sym.o src/bitmap.c nm -a src/bitmap.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/bitmap.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/ckh.sym.o src/ckh.c nm -a src/ckh.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/ckh.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/ctl.sym.o src/ctl.c nm -a src/ctl.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/ctl.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/div.sym.o src/div.c nm -a src/div.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/div.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/extent.sym.o src/extent.c nm -a src/extent.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/extent.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/extent_dss.sym.o src/extent_dss.c nm -a src/extent_dss.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/extent_dss.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/extent_mmap.sym.o src/extent_mmap.c nm -a src/extent_mmap.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/extent_mmap.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/hash.sym.o src/hash.c nm -a src/hash.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/hash.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/hooks.sym.o src/hooks.c nm -a src/hooks.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/hooks.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/large.sym.o src/large.c nm -a src/large.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/large.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/log.sym.o src/log.c nm -a src/log.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/log.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/malloc_io.sym.o src/malloc_io.c nm -a src/malloc_io.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/malloc_io.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/mutex.sym.o src/mutex.c nm -a src/mutex.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/mutex.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/mutex_pool.sym.o src/mutex_pool.c nm -a src/mutex_pool.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/mutex_pool.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/nstime.sym.o src/nstime.c nm -a src/nstime.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/nstime.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/pages.sym.o src/pages.c nm -a src/pages.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/pages.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/prng.sym.o src/prng.c nm -a src/prng.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/prng.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/prof.sym.o src/prof.c nm -a src/prof.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/prof.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/rtree.sym.o src/rtree.c nm -a src/rtree.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/rtree.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/stats.sym.o src/stats.c nm -a src/stats.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/stats.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/sz.sym.o src/sz.c nm -a src/sz.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/sz.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/tcache.sym.o src/tcache.c nm -a src/tcache.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/tcache.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/ticker.sym.o src/ticker.c nm -a src/ticker.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/ticker.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/tsd.sym.o src/tsd.c nm -a src/tsd.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/tsd.sym gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/witness.sym.o src/witness.c nm -a src/witness.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/witness.sym /bin/sh include/jemalloc/internal/private_namespace.sh src/jemalloc.sym src/arena.sym src/background_thread.sym src/base.sym src/bin.sym src/bitmap.sym src/ckh.sym src/ctl.sym src/div.sym src/extent.sym src/extent_dss.sym src/extent_mmap.sym src/hash.sym src/hooks.sym src/large.sym src/log.sym src/malloc_io.sym src/mutex.sym src/mutex_pool.sym src/nstime.sym src/pages.sym src/prng.sym src/prof.sym src/rtree.sym src/stats.sym src/sz.sym src/tcache.sym src/ticker.sym src/tsd.sym src/witness.sym > include/jemalloc/internal/private_namespace.gen.h cp include/jemalloc/internal/private_namespace.gen.h include/jemalloc/internal/private_namespace.gen.h gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/jemalloc.o src/jemalloc.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/arena.o src/arena.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/background_thread.o src/background_thread.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/base.o src/base.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/bin.o src/bin.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/bitmap.o src/bitmap.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/ckh.o src/ckh.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/ctl.o src/ctl.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/div.o src/div.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/extent.o src/extent.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/extent_dss.o src/extent_dss.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/extent_mmap.o src/extent_mmap.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/hash.o src/hash.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/hooks.o src/hooks.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/large.o src/large.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/log.o src/log.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/malloc_io.o src/malloc_io.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/mutex.o src/mutex.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/mutex_pool.o src/mutex_pool.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/nstime.o src/nstime.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/pages.o src/pages.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/prng.o src/prng.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/prof.o src/prof.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/rtree.o src/rtree.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/stats.o src/stats.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/sz.o src/sz.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/tcache.o src/tcache.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/ticker.o src/ticker.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/tsd.o src/tsd.c gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/witness.o src/witness.c g++ -Wall -g3 -fvisibility=hidden -O3 -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/jemalloc_cpp.o src/jemalloc_cpp.cpp ar crus lib/libjemalloc.a src/jemalloc.o src/arena.o src/background_thread.o src/base.o src/bin.o src/bitmap.o src/ckh.o src/ctl.o src/div.o src/extent.o src/extent_dss.o src/extent_mmap.o src/hash.o src/hooks.o src/large.o src/log.o src/malloc_io.o src/mutex.o src/mutex_pool.o src/nstime.o src/pages.o src/prng.o src/prof.o src/rtree.o src/stats.o src/sz.o src/tcache.o src/ticker.o src/tsd.o src/witness.o src/jemalloc_cpp.o ar: `u' modifier ignored since `D' is the default (see `U') make[2]: Leaving directory '/content/milvus/core/milvus/scripts/redis-stable/deps/jemalloc' make[1]: Leaving directory '/content/milvus/core/milvus/scripts/redis-stable/deps' CC adlist.o CC quicklist.o CC ae.o CC anet.o CC dict.o CC server.o CC sds.o CC zmalloc.o CC lzf_c.o CC lzf_d.o CC pqsort.o CC zipmap.o CC sha1.o CC ziplist.o CC release.o CC networking.o CC util.o CC object.o CC db.o CC replication.o CC rdb.o CC t_string.o CC t_list.o CC t_set.o CC t_zset.o CC t_hash.o CC config.o CC aof.o CC pubsub.o CC multi.o CC debug.o CC sort.o CC intset.o CC syncio.o CC cluster.o CC crc16.o CC endianconv.o CC slowlog.o CC scripting.o CC bio.o CC rio.o CC rand.o CC memtest.o CC crcspeed.o CC crc64.o CC bitops.o CC sentinel.o CC notify.o CC setproctitle.o CC blocked.o CC hyperloglog.o CC latency.o CC sparkline.o CC redis-check-rdb.o CC redis-check-aof.o CC geo.o CC lazyfree.o CC module.o CC evict.o CC expire.o CC geohash.o CC geohash_helper.o CC childinfo.o CC defrag.o CC siphash.o CC rax.o CC t_stream.o CC listpack.o CC localtime.o CC lolwut.o CC lolwut5.o CC lolwut6.o CC acl.o CC gopher.o CC tracking.o CC connection.o CC tls.o CC sha256.o CC timeout.o CC setcpuaffinity.o LINK redis-server INSTALL redis-sentinel CC redis-cli.o LINK redis-cli CC redis-benchmark.o LINK redis-benchmark INSTALL redis-check-rdb INSTALL redis-check-aof Hint: It's a good idea to run 'make test' ;) 42370:C 23 Jun 2021 14:54:34.593 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 42370:C 23 Jun 2021 14:54:34.593 # Redis version=6.0.5, bits=64, commit=3fc81236, modified=0, pid=42370, just started 42370:C 23 Jun 2021 14:54:34.593 # Warning: no config file specified, using the default config. In order to specify a config file use ./redis-server /path/to/redis.conf _._ _.-``__ ''-._ _.-`` `. `_. ''-._ Redis 6.0.5 (3fc81236/0) 64 bit .-`` .-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) Running in standalone mode |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379 | `-._ `._ / _.-' | PID: 42370 `-._ `-._ `-./ _.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | http://redis.io `-._ `-._`-.__.-'_.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | `-._ `-._`-.__.-'_.-' _.-' `-._ `-.__.-' _.-' `-._ _.-' `-.__.-' 42370:M 23 Jun 2021 14:54:34.595 # Server initialized 42370:M 23 Jun 2021 14:54:34.595 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. 42370:M 23 Jun 2021 14:54:34.595 * Ready to accept connections 42370:signal-handler (1624462946) Received SIGINT scheduling shutdown... 42370:M 23 Jun 2021 15:42:26.502 # User requested shutdown... 42370:M 23 Jun 2021 15:42:26.502 * Saving the final RDB snapshot before exiting. 42370:M 23 Jun 2021 15:42:26.510 * DB saved on disk 42370:M 23 Jun 2021 15:42:26.510 # Redis is now ready to exit, bye bye...
! nohup ./redis-stable/src/redis-server > redis_nohup.out &
! cat redis_nohup.out
nohup: redirecting stderr to stdout 42581:C 23 Jun 2021 16:02:32.639 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 42581:C 23 Jun 2021 16:02:32.639 # Redis version=6.0.5, bits=64, commit=3fc81236, modified=0, pid=42581, just started 42581:C 23 Jun 2021 16:02:32.639 # Warning: no config file specified, using the default config. In order to specify a config file use ./redis-stable/src/redis-server /path/to/redis.conf 42581:M 23 Jun 2021 16:02:32.641 * Running mode=standalone, port=6379. 42581:M 23 Jun 2021 16:02:32.641 # Server initialized 42581:M 23 Jun 2021 16:02:32.641 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. 42581:M 23 Jun 2021 16:02:32.642 * Ready to accept connections
!pip install -U grpcio
%cd /content
/content
This PaddlePaddle model is used to transform user information into vectors.
!wget https://paddlerec.bj.bcebos.com/aistudio/user_vector.tar.gz --no-check-certificate
!mkdir -p movie_recommender/user_vector_model
!tar xf user_vector.tar.gz -C movie_recommender/user_vector_model/
!rm user_vector.tar.gz
--2021-06-23 16:13:35-- https://paddlerec.bj.bcebos.com/aistudio/user_vector.tar.gz Resolving paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)... 103.235.46.61, 2409:8c00:6c21:10ad:0:ff:b00e:67d Connecting to paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)|103.235.46.61|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 33650924 (32M) [application/x-gzip] Saving to: ‘user_vector.tar.gz’ user_vector.tar.gz 100%[===================>] 32.09M 7.94MB/s in 5.4s 2021-06-23 16:13:42 (5.95 MB/s) - ‘user_vector.tar.gz’ saved [33650924/33650924]
Downloading Data
# Download movie information
!wget -P movie_recommender https://paddlerec.bj.bcebos.com/aistudio/movies.dat --no-check-certificate
# Download movie vecotrs
!wget -P movie_recommender https://paddlerec.bj.bcebos.com/aistudio/movie_vectors.txt --no-check-certificate
--2021-06-23 16:13:43-- https://paddlerec.bj.bcebos.com/aistudio/movies.dat Resolving paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)... 103.235.46.61, 2409:8c00:6c21:10ad:0:ff:b00e:67d Connecting to paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)|103.235.46.61|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 171308 (167K) [application/octet-stream] Saving to: ‘movie_recommender/movies.dat’ movies.dat 100%[===================>] 167.29K 167KB/s in 1.0s 2021-06-23 16:13:46 (167 KB/s) - ‘movie_recommender/movies.dat’ saved [171308/171308] --2021-06-23 16:13:46-- https://paddlerec.bj.bcebos.com/aistudio/movie_vectors.txt Resolving paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)... 103.235.46.61, 2409:8c00:6c21:10ad:0:ff:b00e:67d Connecting to paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)|103.235.46.61|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1095505 (1.0M) [text/plain] Saving to: ‘movie_recommender/movie_vectors.txt’ movie_vectors.txt 100%[===================>] 1.04M 648KB/s in 1.7s 2021-06-23 16:13:49 (648 KB/s) - ‘movie_recommender/movie_vectors.txt’ saved [1095505/1095505]
Importing Movies into Milvus
! lsof -i -P -n | grep -E 'milvus|redis'
milvus_se 42433 root 17u IPv4 478871 0t0 TCP *:19121 (LISTEN) milvus_se 42433 root 20u IPv4 479283 0t0 TCP *:19530 (LISTEN) redis-ser 42581 root 6u IPv6 507112 0t0 TCP *:6379 (LISTEN) redis-ser 42581 root 7u IPv4 507113 0t0 TCP *:6379 (LISTEN)
from milvus import Milvus, IndexType, MetricType, Status
import redis
milv = Milvus(host = '127.0.0.1', port = 19530)
r = redis.StrictRedis(host="127.0.0.1", port=6379)
milv.client_version()
'1.1.0'
We begin by loading all the movie files into redis.
import json
import codecs
#1::Toy Story (1995)::Animation|Children's|Comedy
def process_movie(lines, redis_cli):
for line in lines:
if len(line.strip()) == 0:
continue
tmp = line.strip().split("::")
movie_id = tmp[0]
title = tmp[1]
genre_group = tmp[2]
tmp = genre_group.strip().split("|")
genre = tmp
movie_info = {"movie_id" : movie_id,
"title" : title,
"genre" : genre
}
redis_cli.set("{}##movie_info".format(movie_id), json.dumps(movie_info))
with codecs.open("movie_recommender/movies.dat", "r",encoding='utf-8',errors='ignore') as f:
lines = f.readlines()
process_movie(lines, r)
COLLECTION_NAME = 'demo_films'
PARTITION_NAME = 'Movie'
#Dropping collection for clean slate run
milv.drop_collection(COLLECTION_NAME)
param = {'collection_name':COLLECTION_NAME,
'dimension':32,
'index_file_size':2048,
'metric_type':MetricType.L2
}
milv.create_collection(param)
# milv.create_partition(COLLECTION_NAME, PARTITION_NAME)
Status(code=0, message='Create collection successfully!')
milv.get_collection_info(COLLECTION_NAME)
(Status(code=0, message='Describe collection successfully!'), CollectionSchema(collection_name='demo_films', dimension=32, index_file_size=2048, metric_type=<MetricType: L2>))
The vectors in movie_vectors.txt
are obtained from the user_vector_model
downloaded above. So we can directly get the vectors and the IDs by reading the file.
def get_vectors():
with codecs.open("movie_recommender/movie_vectors.txt", "r", encoding='utf-8', errors='ignore') as f:
lines = f.readlines()
ids = [int(line.split(":")[0]) for line in lines]
embeddings = []
for line in lines:
line = line.strip().split(":")[1][1:-1]
str_nums = line.split(",")
emb = [float(x) for x in str_nums]
embeddings.append(emb)
return ids, embeddings
ids, embeddings = get_vectors()
Import vectors into the partition Movie under the collection demo_films.
# status = milv.insert(collection_name=COLLECTION_NAME, records=embeddings, ids=ids, partition_tag=PARTITION_NAME)
status = milv.insert(collection_name=COLLECTION_NAME, records=embeddings, ids=ids)
status[0]
Status(code=0, message='Add vectors successfully!')
Pass in the gender, age and occupation of the user we want to recommend. user_vector_model model will generate the corresponding user vector. Occupation is chosen from the following choices:
import numpy as np
from paddle_serving_app.local_predict import LocalPredictor
class RecallServerServicer(object):
def __init__(self):
self.uv_client = LocalPredictor()
self.uv_client.load_model_config("movie_recommender/user_vector_model/serving_server_dir")
def hash2(self, a):
return hash(a) % 1000000
def get_user_vector(self):
dic = {"userid": [], "gender": [], "age": [], "occupation": []}
lod = [0]
dic["userid"].append(self.hash2('0'))
dic["gender"].append(self.hash2('M'))
dic["age"].append(self.hash2('23'))
dic["occupation"].append(self.hash2('6'))
lod.append(1)
dic["userid.lod"] = lod
dic["gender.lod"] = lod
dic["age.lod"] = lod
dic["occupation.lod"] = lod
for key in dic:
dic[key] = np.array(dic[key]).astype(np.int64).reshape(len(dic[key]),1)
fetch_map = self.uv_client.predict(feed=dic, fetch=["save_infer_model/scale_0.tmp_1"], batch=True)
return fetch_map["save_infer_model/scale_0.tmp_1"].tolist()[0]
recall = RecallServerServicer()
user_vector = recall.get_user_vector()
2021-06-23 16:29:24,262 - INFO - LocalPredictor load_model_config params: model_path:movie_recommender/user_vector_model/serving_server_dir, use_gpu:False, gpu_id:0, use_profile:False, thread_num:1, mem_optim:True, ir_optim:False, use_trt:False, use_lite:False, use_xpu: False, use_feed_fetch_ops:False
user_vector
[0.0, 4.911433696746826, 4.132595062255859, 3.2255895137786865, 0.0, 4.944108963012695, 0.0, 0.0, 1.27165687084198, 3.1072912216186523, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.9184402227401733, 0.0, 0.0, 0.0, 4.42396354675293, 2.0686450004577637, 0.0]
Pass in the user vector, and then recall vectors in the previously imported data collection and partition.
TOP_K = 20
SEARCH_PARAM = {'nprobe': 20}
status, results = milv.search(collection_name=COLLECTION_NAME, query_records=[user_vector], top_k=TOP_K, params=SEARCH_PARAM)
recall_results = []
for x in results[0]:
recall_results.append(r.get("{}##movie_info".format(x.id)).decode('utf-8'))
recall_results
['{"movie_id": "760", "title": "Stalingrad (1993)", "genre": ["War"]}', '{"movie_id": "1350", "title": "Omen, The (1976)", "genre": ["Horror"]}', '{"movie_id": "1258", "title": "Shining, The (1980)", "genre": ["Horror"]}', '{"movie_id": "632", "title": "Land and Freedom (Tierra y libertad) (1995)", "genre": ["War"]}', '{"movie_id": "3007", "title": "American Movie (1999)", "genre": ["Documentary"]}', '{"movie_id": "2086", "title": "One Magic Christmas (1985)", "genre": ["Drama", "Fantasy"]}', '{"movie_id": "1051", "title": "Trees Lounge (1996)", "genre": ["Drama"]}', '{"movie_id": "3920", "title": "Faraway, So Close (In Weiter Ferne, So Nah!) (1993)", "genre": ["Drama", "Fantasy"]}', '{"movie_id": "1303", "title": "Man Who Would Be King, The (1975)", "genre": ["Adventure"]}', '{"movie_id": "652", "title": "301, 302 (1995)", "genre": ["Mystery"]}', '{"movie_id": "1605", "title": "Excess Baggage (1997)", "genre": ["Adventure", "Romance"]}', '{"movie_id": "1275", "title": "Highlander (1986)", "genre": ["Action", "Adventure"]}', '{"movie_id": "1126", "title": "Drop Dead Fred (1991)", "genre": ["Comedy", "Fantasy"]}', '{"movie_id": "792", "title": "Hungarian Fairy Tale, A (1987)", "genre": ["Fantasy"]}', '{"movie_id": "2228", "title": "Mountain Eagle, The (1926)", "genre": ["Drama"]}', '{"movie_id": "2659", "title": "It Came from Hollywood (1982)", "genre": ["Comedy", "Documentary"]}', '{"movie_id": "2545", "title": "Relax... It\'s Just Sex (1998)", "genre": ["Comedy"]}', '{"movie_id": "1289", "title": "Koyaanisqatsi (1983)", "genre": ["Documentary", "War"]}', '{"movie_id": "2537", "title": "Beyond the Poseidon Adventure (1979)", "genre": ["Adventure"]}', '{"movie_id": "2864", "title": "Splendor (1999)", "genre": ["Comedy"]}']
After completing the recall service, the results can be further sorted using the movie_recommender model, and then the movies with high similarity scores can be recommended to users. You can try this deployable recommendation system using this quick start.