数据可视化如果做到以下几点就属于比较好的可视化平台了
1)过程简单、生产效率要高(无论是研发、运营、产品都快可以快速的创建想要的图表)。
2)自定义程度高:展现形式多样可选、交互自定义程度高(各种筛选过滤)。
3)便于分享:数据的分享是数据价值体现的一个环节。
4)数据的评论和备注:备注能够记录异常和原因,评论利于保存讨论和分析过程(一个相对完善的系统而言这个功能方便实际使用)。
5)权限控制:自由总是相对的,相对分享就是控制,张弛有度必是这个系统长存的基石
实现上述的可视化系统并非易事,里面涉及到很多的环节。
但可视化做的比较好的tableau却给我们创建了一个比较好的典范。它在数据的分享、效率方面做得无可挑剔。
实际环境中却不一定能直接使用一些成熟的解决方案,还需要自己搭建。

继续阅读

  • es中文社区
    1. https://github.com/elasticsearch-cn
  • es权威指南
    1. https://es.xiaoleilu.com/
      https://elasticsearch.cn/book/elasticsearch_definitive_guide_2.x/
  • 搜索类型
      es在查询时,可以指定搜索类型为QUERY_THEN_FETCH,QUERY_AND_FEATCH,DFS_QUERY_THEN_FEATCH和DFS_QUERY_AND_FEATCH
      http://www.cnblogs.com/donlianli/p/3857500.html
      https://www.elastic.co/blog/understanding-query-then-fetch-vs-dfs-query-then-fetch
  • 常用插件:
    1. sql
      如果查询不会写可以用这个 看看他的dsl
      https://github.com/NLPchina/elasticsearch-sql
      header
      https://github.com/mobz/elasticsearch-head
  • 相关资料
    1. exploring elasticsearch http://exploringelasticsearch.com/github_interview.html
      #研发解决方案介绍#基于ES的搜索+筛选+排序解决方案 https://www.cnblogs.com/zhengyun_ustc/p/55solution6.html
  • hadoop集成
    1. https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html
  • hive集成
    1. https://www.elastic.co/guide/en/elasticsearch/hadoop/current/hive.html
      https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-hadoop/5.6.11
    常用命令:
    查看索引
    http://127.0.0.1:9200/_cat/indices?v

    curl -i -XGET 'http://127.0.0.1:9200/_count?pretty'

    新增数据
    curl -i -XPOST 'http://127.0.0.1:9200/com/employee/1' -d'{"first_name":"John","last_name":"Smith","age":25,"about":"i love go rock climbing","interests":["sport","music"]}'
    curl -i -XGET 'http://127.0.0.1:9200/com/employee/1?pretty'

    curl -i -XPOST 'http://127.0.0.1:9200/com/employee/2' -d'{"first_name":"weimingwei","last_name":"mingming","age":20,"about":"i love go rock climbing","interests":["music"]}'
    curl -i -XPOST 'http://127.0.0.1:9200/com/employee/3' -d'{"first_name":"yang","last_name":"wenshuai","age":60,"about":"good good study day day up ","interests":["painting"]}'
    curl -i -XPOST 'http://127.0.0.1:9200/com/employee/4' -d'{"first_name":"66777","last_name":"324234324","age":60,"about":"good good study day day up ","interests":["painting"]}'

    简单搜索:
    curl -i -XGET 'http://127.0.0.1:9200/com/employee/_search?pretty'
    curl -i -XGET 'http://127.0.0.1:9200/com/employee/_search?q=last_name=wenshuai&pretty'
    复杂查询
    curl -i -XGET 'http://127.0.0.1:9200/com/employee/_search' -d'
    {
    "query": {
    "filtered": {
    "filter": {
    "range": {
    "age": {
    "gt": 30
    }
    }
    },
    "query": {
    "match": {
    "last_name": "mingwei"
    }
    }
    }
    }
    }'

    全文搜索
    curl -i -XGET 'http://127.0.0.1:9200/com/employee/_search?pretty' -d'
    {

    "query": {
    "match": {
    "about": "love"
    }
    }

    }'

    curl -i -XPUT 'http://127.0.0.1:9200/twitter/_settings?pretty' -d'{"number_of_shards":1,"number_of_replicas":2}'

    任意往里面放数据
    curl -i -XPUT 'http://127.0.0.1:9200/web/nn/123?pretty' -d'{"number_of_shards":1,"number_of_replicas":2}'
    curl -XGET 'http://127.0.0.1:9200/web/nn/123?pretty'
    {
    "_index" : "web",
    "_type" : "nn",
    "_id" : "123",
    "_version" : 1,
    "found" : true,
    "_source" : {
    "number_of_shards" : 1,
    "number_of_replicas" : 2
    }

    过滤字段获取
    curl -XGET 'http://127.0.0.1:9200/web/nn/123?_source=number_of_shards&pretty'
    {
    "_index" : "web",
    "_type" : "nn",
    "_id" : "123",
    "_version" : 1,
    "found" : true,
    "_source" : {
    "number_of_shards" : 1
    }
    只要source
    curl -XGET 'http://127.0.0.1:9200/web/nn/123/_source?pretty'
    {
    "number_of_shards" : 1,
    "number_of_replicas" : 2
    }

    检查是否存在 curl -i -XHEAD 'http://127.0.0.1:9200/web/nn/133' -I

    删除
    curl -XPOST http://10.0.12.108:6335/app_subjoin_renewal_data_di/app_subjoin_renewal_data_di/_delete_by_query -d'
    {"query":{"bool":{"filter":[{"match_phrase":{"date":20180131}}]}}}'

    删除所有:
    curl -XPOST '10.0.9.108:9200/homework/minactivtyfinish/_delete_by_query?conflicts=proceed&pretty' -d'
    {
    "query": {
    "match_all": {}
    }
    }'

    curl -XPOST 'http://10.0.12.110:6335/app_service_tms_teacher_statics_da/app_service_tms_teacher_statics_da/_delete_by_query?conflicts=proceed&pretty' -d'{"query": {"range" : "update_time" : {
    "lt" :
    }
    }
    }
    }'

    中文问题
    curl -i -XPOST 'http://127.0.0.1:9200/com/employee/1' -d'{"first_name":"王伟伟","last_name":"王雪峰","age":25,"about":"i love go rock climbing","interests":["sport","music"]}'

    curl 'http://127.0.0.1:9200/com/employee/_search?pretty' -d'
    {
    "query": {
    "prefix" : { "first_name" : "王伟" }
    }
    }'

    curl 'http://127.0.0.1:9200/com/employee/_search?pretty' -d'
    {
    "query": {
    "prefix" : { "last_name" : "王" }
    }
    }'

    curl -XGET 'http://10.0.12.110:6335/app_es_rpt_kpi_teacher_structure_di/app_es_rpt_kpi_teacher_structure_di/_search?pretty'

    curl 'http://127.0.0.1:9200/com/employee/_search?pretty' -d'
    { "query": {
    "prefix" : { "last_name" : "wen" }
    }
    }

    '

    #scroll
    curl http://10.1.1.246:9200/app_runup_new_teacher_data_di/app_runup_new_teacher_data_di/_search?scroll=1m -d'
    {
    "query": { "match_all": {}},
    "sort" : ["_doc"], //the most efficient sort order
    "size": 10
    }'

    curl http://10.1.1.246:9200/_search/scroll -d'
    {
    "scroll": "1m",
    "scroll_id" : "DnF1ZXJ5VGhlbkZldGNoBQAAAAAACzEGFnI2WFh1RDVJVEt5RERKZ0xhRXZnckEAAAAAAAsxBxZyNlhYdUQ1SVRLeURESmdMYUV2Z3JBAAAAAAALMQgWcjZYWHVENUlUS3lEREpnTGFFdmdyQQAAAAAACzEKFnI2WFh1RDVJVEt5RERKZ0xhRXZnckEAAAAAAAsxCRZyNlhYdUQ1SVRLeURESmdMYUV2Z3JB"
    }'

    删除索引

    curl -XDELETE '10.0.12.108:6335/bi_app_subjoin_renewal_data_flag_di?pretty'
    curl -XDELETE '10.0.12.108:6335/bi_app_subjoin_renewal_data_di?pretty'

    curl -XDELETE '10.0.12.108:6335/app_subjoin_renewal_data_flag_di?pretty'
    curl -XDELETE '10.0.12.108:6335/app_subjoin_renewal_data_di?pretty'