张聪《ngx_lua在又拍云的应用：日志收集及作用》PPT原版.pptx

Monkey Zhang (timebug)2016.07 @ Shenzhen ArchSummit日志收集、性能调优与服务化ngx_lua 在又拍云的应用A Systems Engineer at UPYUNEmail: mailto:@@Github: /timebug/timebug$ ./configure --prefix=/opt/nginx \ --add-module=/path/to/lua-nginx-modulehttp { server { listen 8080; location /add { set $res ; rewrite_by_lua local a = tonumber(ngx.var.arg_a) or 0 local b = tonumber(ngx.var.arg_b) or 0 ngx.var.res = a + b ; content_by_lua ngx.say(ngx.var.res) ; } }}$ curl http://localhost:8080/add?a=6b=713LuaJITLuaJIT is a Just-In-Time Compiler (JIT) for the Lua programming language.How it worksLuaJIT VM embedded into the Nginxcosocket APIngx.socket.* connect send receive sslhandshake close settimeout etc.UPYUN CDN API is built on top of NGINX with ngx_luaUPYUN APIBase64 Filter by Luahttp { lua_package_path “$prefix/app/src/?.lua;;; server { listen 8080; location /base64 { set $b64_en ; set $b64_e0 ; set $b64_e1 ; echo_duplicate 1000 hello; header_filter_by_lua ngx.header.content_length = nil -- ((n + 2) / 3 ) * 4 ngx.header.content_type = text/plain ngx.header.content_transfer_encoding = base64 ; body_filter_by_lua_file app/src/b64_body_filter.lua; } }}Base64 Filter by Lua:Chunk by Chunklocal chunk = ngx.arg[1]local e0 = ngx.var.b64_e0 or local e1 = ngx.var.b64_e1 or local en = tonumber(ngx.var.b64_en) or 0if en == 1 then chunk = e0 .. chunkelseif en == 2 then chunk = e0 .. e1 .. chunkendif not ngx.arg[2] then en = #chunk % 3 if en == 1 then e0 = chunk:sub(-1) elseif en == 2 then e1 = chunk:sub(-1) e0 = chunk:sub(-2, -2) end chunk = chunk:sub(1, #chunk - en)else -- eof en = 0endngx.var.b64_en = enngx.var.b64_e0 = e0ngx.var.b64_e1 = e1ngx.arg[1] = ngx.encode_base64(chunk)location /upload { proxy_request_buffering off; . . .}Lua Streaming Uploadngx.req.init_body()ngx.req.append_body(chunk)ngx.req.finish_body()UPYUN CDN全网 CDN 日志规模有多大？ 保守估计每天约 5 ~ 10T 原始日志。日志的多种用途日志每日归档；按不同服务名称提供下载。日志实时分析；供近实时多维分析、问题排查和监控报警。日志聚合计算；数据结果提前根据需求确定，实时后台展示。日志离线分析；复杂的数据模型计算和分析，定期生成报表。日志收集常见的几种方式access_log /path/to/access.log