部署验证OpenGrok

How to setup OpenGrok

How to setup OpenGrok

在用户目录下执行,仅作验证使用。

安装准备

  •  从https://github.com/oracle/opengrok/releases下载压缩包,目前最新的为opengrok-1.3.16.tar.gz
  • 创建目录 mkdir -p opengrok/{src,data,dist,etc,log} src目录作为源码仓库,data目录作为indexer目录
  • 解压压缩包tar -C opengrok/dist --strip-components=1 -xzf opengrok-1.3.16.tar.gz (–strip-components=1 参数Strip NUMBER leading components from file names on extraction。 将移除一层目录)
  • 指定log配置cp opengrok/dist/doc/logging.properties opengrok/etc
  • 在src目录下git clone几个代码仓库

ubuntu下安装universal-ctags

官方教程

  • 安装依赖环境
sudo apt install \
    gcc make \
    pkg-config autoconf automake \
    python3-docutils \
    libseccomp-dev \
    libjansson-dev \
    libyaml-dev \
    libxml2-dev
  • 拉取代码git clone https://github.com/universal-ctags/ctags.git
  • 在ctags目录下,执行./autogen.sh./configuremakesudo make install

成功后,默认ctags位于 /usr/local/bin/ctags

$ ./autogen.sh
$ ./configure --prefix=/where/you/want # defaults to /usr/local
$ make
$ make install # may require extra privileges depending on where to install
  • 生成配置文件
java \
    -Djava.util.logging.config.file=/opengrok/etc/logging.properties \
    -jar opengrok/dist/lib/opengrok.jar \
    -c /usr/local/bin/ctags \
    -s opengrok/src -d opengrok/data -H -P -S -G \
    -W opengrok/etc/configuration.xml 
  • 复制opengrok/dist/lib/目录下的source.war文件到tomcat服务器的webapps目录

安装管理工具(可选)

python包包括了一下封装后的命令,包括indexer等。在release版本的tar包解压后的tools文件夹下包含opengrok-tools.tar.gz文件,使用pip命令安装此文件即可。

  • 先确保有python3环境
  • 安装python3环境下的pip sudo apt install python3-pip(查看帮助pip3 --help,或者pip3 install --help), 检查版本pip3 --version pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)
  • 到tool目录下执行pip3 install opengrok-tools.tar.gz

OpenGrok tools

  • 确保有python3环境,最好使用独立的虚拟环境。创建虚拟环境python3 -m venv opengroktools (ubuntu环境下需要先安装 apt-get install python3-venv
  • 将虚拟环境中的pip升级到最新版本opengroktools/bin/python -m pip install --upgrage pip (否则在安装opengrok-tools.tar.gz时可能会报错)
  • 安装工具包opengroktools/bin/python -m pip install /tools/opengrok-tools.tar.gz

在单位虚拟环境目录下可以使用这些tools,例如opengroktools/bin/opengrok-indexer

激活与失效展示:

#使用当前虚拟环境
➜  ~ . opengroktools/bin/activate
(opengroktools) ➜  ~ opengrok-indexer --help
usage: opengrok-indexer [-h] [-l LOGLEVEL] [-v] [-j JAVA] [-J JAVA_OPTS]
                        [-e ENVIRONMENT] [--doprint boolean]
                        (-a JAR | -c CLASSPATH) [-C]
                        options [options ...]

OpenGrok indexer wrapper

positional arguments:
  options               options

optional arguments:
  -h, --help            show this help message and exit
  -l LOGLEVEL, --loglevel LOGLEVEL
                        Set log level (e.g. "ERROR")
  -v, --version         Version of the tool
  -j JAVA, --java JAVA  path to java binary
  -J JAVA_OPTS, --java_opts JAVA_OPTS
                        java options. Use one for every java option, e.g.
                        -J=-server -J=-Xmx16g
  -e ENVIRONMENT, --environment ENVIRONMENT
                        Environment variables in the form of name=value
  --doprint boolean     Enable/disable printing of messages from the
                        application as they are produced.
  -a JAR, --jar JAR     Path to jar archive to run
  -c CLASSPATH, --classpath CLASSPATH
                        Class path
  -C, --no_ctags_check  Suppress checking for ctags
# 退出虚拟环境
(opengroktools) ➜  ~ deactivate
➜  ~
# 使用命令
➜  ~ opengrok-index --help
zsh: command not found: opengrok-index
# 使用绝对路径
➜  ~ opengroktools/bin/opengrok-indexer --help
usage: opengrok-indexer [-h] [-l LOGLEVEL] [-v] [-j JAVA] [-J JAVA_OPTS]
                        [-e ENVIRONMENT] [--doprint boolean]
                        (-a JAR | -c CLASSPATH) [-C]
                        options [options ...]
...

暴力删除虚拟环境: rm -r opengroktools

安装web服务器

Setup Apache Tomcat 8 | 8.5 on Ubuntu 16.04 | 18.04 LTS

  • 下载压缩包,解压压缩包,指定端口号,直接到bin目录执行./catalina.sh start 将启动默认的服务器
  • 修改source工程(即openGrok服务)的配置文件tomcat8/webapps/source/WEB-INF/web.xml,指定服务变量 CONFIGURATION的值
<context-param>
  <description>Full path to the configuration file where OpenGrok can read its configuration</description>
  <param-name>CONFIGURATION</param-name>
  <param-value>/home/xxx/opengrok/etc/configuration.xml</param-value>
</context-param>

搜索方法

OpenGrok 搜索用法

  • 主页的Help按钮包含了基础的使用方法
  • 官方API调用说明文档:opengrop API on apiary(Powerful API Design Stack. Built for Developers)
  • openGrok底层使用lucene,支持的正则表达式参考Lucene regexp page

  • Full Search (full) Search through all text tokens (words,strings,identifiers,numbers) in index.

  • Definition (defs) Only finds symbol definitions (where e.g. a variable (function, …) is defined).

  • Symbol (refs) Only finds symbols (e.g. methods, classes, functions, variables).

  • File Path (path) path of the source file (no need to use dividers, or if, then use “/” - Windows users, “\” is an escape key in Lucene query syntax!
    Please don’t use “\“, or replace it with “/”).
    Also note that if you want just exact path, enclose it in “”, e.g. “src/mypath”, otherwise dividers will be removed and you get more hits.

  • History (hist) History log comments.

23 Google Search Tips You’ll Want to Learn
https://www.google.com/advanced_search
https://www.google.com/advanced_image_search
搜索操作符支持,类似的google支持的Search operators

gitlab API 获取project

Gitlab : List all the projects and all the groups

分别调用projects接口和groups接口即可,类似——

curl --head "https://<host/api/v4/projects?private_token=<your private token>&per_page=100&page=<page_number>"

只返回head部分,其中包括X-TotalX-Total-Pages的值。默认page从1开始计算,默认每页返回20条,最大返回100个条目

官方推荐的gitlab api客户端实现

How to retrieve repository size through REST API

"statistics": {
      "commit_count": 5914,
      "storage_size": 1727206,
      "repository_size": 0,
      "wiki_size": 52428,
      "lfs_objects_size": 0,
      "job_artifacts_size": 1674778
    },

The size is listed in Bytes. To convert to KB, divide by 1,000. To convert to MB, divide by 1,000,000.

How can I see the size of a GitHub repository before cloning it?

There’s a way to access this information through the GitHub API.

When retrieving information about a repository, a property named size is valued with the size of the whole repository (including all of its history), in kilobytes.

For instance, the Git repository weights around 124 MB. The size property of the returned JSON payload is valued to 124283.

403 Access Denied on Tomcat 8 Manager App without prompting for user/password

Manager App HOW-TO

centOS 上可能遇到的问题

  • 依赖环境的参数不同
# on Centos: diff with Fedora on python3-docutils vs. python3-docutils-doc
sudo yum install -y \
    gcc make \
    pkgconfig autoconf automake \
    python3-docutils-doc \
    libseccomp-devel \
    jansson-devel \
    libyaml-devel \
    libxml2-devel  

17:23:31 WARNING: GitRepository not working (missing binaries?): xxx
17:23:31 WARNING: Failed to determineCurrentVersion for xxx: java.io.IOException: fatal: unknown date format iso8601-strict

Install Latest Git ( Git 2.x ) on CentOS 7

  1. 卸载旧git sudo yum remove git*
  2. 添加endpoint源sudo yum -y install https://packages.endpoint.com/rhel/7/os/x86_64/endpoint-repo-1.7-1.x86_64.rpm
  3. 安装新git sudo yum install git
  4. 检查版本: git version 2.24.1

opengrok tools使用

Per project management and workflow

OpenGrok项目管理 最后使用 opengrok-indexer时认为最后两个参数被忽略了——

opengroktools/bin/opengrok-indexer -a opengrok/dist/lib/opengrok.jar -- \
    -c /usr/local/bin/ctags \
    -U 'http://localhost:8080/source' \
    -s /data1/data/groups -d opengrok/data0907 \
    -R fresh_config.xml
    -H 234_sale \
    234_sale

事实上,opengrok-indexer执行当个项目的index动作时,不需要指定源码、数据目录。这些值在配置文件-R fresh_config.xml中已经定义。使用官方的方法即可,效果如下——

opengroktools/bin/opengrok-indexer -a opengrok/dist/lib/opengrok.jar -- \
>     -c /usr/local/bin/ctags \
>     -U 'http://localhost:8080/source' \
>     -R fresh_config.xml
    -H sub1 \
    sub1 Sep 11, 2020 3:59:44 PM org.opengrok.indexer.configuration.Configuration read
INFO: Reading configuration from /home/vip/fresh_config.xml
Sep 11, 2020 3:59:45 PM org.opengrok.indexer.index.Indexer parseOptions
...
..
.
INFO: Starting traversal of directory /t24_2650_ai-cv-ocr
Sep 11, 2020 4:20:11 PM org.opengrok.indexer.util.Statistics report
INFO: Done traversal of directory /t24_2650_ai-cv-ocr (took 33.529 seconds)
Sep 11, 2020 4:20:11 PM org.opengrok.indexer.index.IndexDatabase update
INFO: Starting indexing of directory /t24_2650_ai-cv-ocr
Sep 11, 2020 4:20:11 PM org.opengrok.indexer.util.Statistics report
INFO: Done indexing of directory /t24_2650_ai-cv-ocr (took 0 ms)
Sep 11, 2020 4:20:12 PM org.opengrok.indexer.index.IndexDatabase update
INFO: Starting traversal of directory /t34_69_public_service

-H参数实际是由opengrok.jar包提供的,详见下。这样默认会将log输出到控制台,等待结果中,看起来新添加项目后,还需要对已有项目“巡视”一番?

opengroktools/bin/opengrok-projadm -b opengrok -d 342_znkf删除效果——

$ opengroktools/bin/opengrok-projadm -b opengrok -d 342_znkf
Deleting project 342_znkf and its index data
Removing source code under /data1/data/groups/342_znkf
Refreshing configuration

将删除索引文件、源码文件、刷新配置文件。

使用curl -s -X GET http://localhost:8080/source/api/v1/configuration -o fresh_config.xml 可以下载更新后的配置文件。可以看到已不再包含刚删除的项目。

 
comments powered by Disqus