MediaWiki Tools

From Evgeny Goldin

Jump to: navigation, search

While working with this Wiki I developed a number of Groovy scripts to keep all pages organized. Those scripts are not versioned, they work for me but they may not work for you - we may have different needs and input files.

Still, I'd love to hear whenever it happens.


Contents

"Links" - Verifies MediaWiki links

links.groovy verifies all internal and cross-page links in MediaWiki files are correct.

  • Internal links are those of form [[#Usage|link text]] where "Usage" is an existing section in the same page.
  • Cross-page links are those of form "http://wiki/MyPage#Section" where "Section" is an existing section in the referenced "MyPage".


Usage

groovy links.groovy <directory> [files include pattern]


  • If no include pattern is provided then all files in directory specified are assumed to be MediaWiki pages.
  • Include pattern is the one recognized by Ant's DirectoryScanner.
  • External links are links starting with "http" and having a "#" as part of the URL. For "http://wiki/MyPage#Section" the link is verified only if "MyPage.anything" file was found when scanning for files.
  • Execution fails if broken links are found.


Example execution (taken from this test verifying this Wiki links):

$ groovy links.groovy /Users/evgenyg/Projects/wiki/ '**/*.txt'
 
...
[/Users/evgenyg/Projects/wiki/maven-plugins/assert-maven-plugin.txt]       - 0   internal links
[/Users/evgenyg/Projects/wiki/maven-plugins/copy-maven-plugin.txt]         - 47  internal links
[/Users/evgenyg/Projects/wiki/maven-plugins/find-maven-plugin.txt]         - 0   internal links
[/Users/evgenyg/Projects/wiki/maven-plugins/jenkis-maven-plugin.txt]       - 4   internal links
...
[/Users/evgenyg/Projects/wiki/gcommons/Main.txt]                           - 116 external links
[/Users/evgenyg/Projects/wiki/gradle-plugins/Gradle-CodeNarc-plugin.txt]   - 15  external links
[/Users/evgenyg/Projects/wiki/gradle-plugins/Gradle-about-plugin.txt]      - 16  external links
[/Users/evgenyg/Projects/wiki/gradle-plugins/Gradle-duplicates-plugin.txt] - 15  external links
[/Users/evgenyg/Projects/wiki/gradle-plugins/Main.txt]                     - 9   external links
[/Users/evgenyg/Projects/wiki/maven-plugins/Duplicates-finder-plugin.txt]  - 17  external links
[/Users/evgenyg/Projects/wiki/maven-plugins/Main.txt]                      - 194 external links
...
[39] files processed (486 ms)


"Spaces" - Fixes MeddiaWiki pages spacing

spaces.groovy updates MediaWiki files so that every "= Section =" is preceded by two empty lines at least. It also trims all trailing spaces in each line.


Usage

groovy spaces.groovy <directory> [files include pattern] [fail if update]


  • If no include pattern is provided then all files in directory specified are assumed to be MediaWiki pages.
  • Include pattern is the one recognized by Ant's DirectoryScanner.
  • If last argument is true then execution fails whenever any page was updated. It helps to verify all pages have the correct spacing set.


Example execution (taken from this test verifying this Wiki pages):

$ groovy spaces.groovy /Users/evgenyg/Projects/wiki/ '**/*.txt'
Fixing spaces in [/Users/evgenyg/Projects/wiki]/[**/*.txt], fail if update [false]
[/Users/evgenyg/Projects/wiki/MediaWiki Tools/Main.txt] updated
Fixing spaces in [/Users/evgenyg/Projects/wiki]/[**/*.txt], fail if update [false] - Done (587 ms)
$ groovy spaces.groovy /Users/evgenyg/Projects/wiki/ '**/*.txt' true
Fixing spaces in [/Users/evgenyg/Projects/wiki]/[**/*.txt], fail if update [true]
Fixing spaces in [/Users/evgenyg/Projects/wiki]/[**/*.txt], fail if update [true] - Done (593 ms)


"g2m" - Converts GitHub flavored Markdown pages to MediaWiki

g2m.groovy converts GitHub flavored Markdown files to MediaWiki syntax. It helps to convert "README.markdown" files and make them as close as possible to be used in MediaWiki.

For example, Spock extensions page in this Wiki is a converted version of the corresponding "README.markdown" with minor manual edits applied.


Usage

groovy g2m.groovy <markdown file> <output file>


"y2m" - Converts YouTrack CSV export files to MediaWiki

y2m.groovy translates YouTrack CSV exports into MediaWiki tables.


Issue Id Type Summary
gc-86 Bug Ant scanner fix - set basedir with canonical path
gc-78 Bug Duplicate log messages
gc-40 Feature FileBean.checksum() - add checksum support
gc-90 Feature FileBean.copyDir() - copy directory recursively
gc-77 Feature FileBean.pack() - add "fullpath" and "prefix" options
gc-85 Feature FileBean.pack() - allow to optionally use TrueZip when packing archives
gc-93 Feature FileBean.pack() - support "filemode" specified as part of include pattern
gc-88 Feature FileBean.pack() - support "fullpath", "prefix" for all archives (zip/tar/tar.gz) and "filemode" for tar/tar.gz
gc-84 Feature FileBean.unpack() - allow to optionally use TrueZip when unpacking archives
gc-87 Feature GeneralBean.executeWithResult() returning String (stdout + stderr)
gc-89 Feature Treat "sar" extension as zip archive
gc-95 Feature VerifyBean.equal() - provide an option whether or not to fail if no files were found
gc-94 Feature VerifyBean.equal() - use combination "directory/pattern" to match files
gc-67 Usability Problem Do not impose logging backend


Usage

If you run the script without any arguments it displays the usage:

$ groovy y2m.groovy
 
------------------------------------------------------------------------------------------------------------------------------
Usage: groovy y2m.groovy <YouTrack URL> <CSV file> [<Fields>] [<Group-By-Fields>] [<Add #>]
------------------------------------------------------------------------------------------------------------------------------
YouTrack URL    - Base URL of YouTrack application, like "http://youtrack.jetbrains.net" or "http://evgeny-goldin.org/youtrack"
CSV file        - "Issues in CSV"-exported file from YouTrack
Fields          - (optional) comma-separated list of fields to use in MediaWiki table, "${ defaultFields.join( ', ' )}" by default
Group-By-Fields - (optional) comma-separated list of fields to group table rows by, "${ defaultGroupByFields.join( ', ' )}" for the default fields
Add #           - (optional) true/false, whether to add a '#' column with a running counter for each issue, false by default
 
If -Dy2mFile=file is specifed then result is written to this file instead of writing it to stdout.
------------------------------------------------------------------------------------------------------------------------------


The table above was generated by running:

groovy y2m.groovy http://evgeny-goldin.org/youtrack export.csv > 1.txt


Since Groovy 1.8.3 allows to run scripts retrieved over HTTP I have the following alias defined in my "~/.bash_profile":

alias y2m='groovy https://raw.github.com/evgeny-goldin/scripts/master/src/main/groovy/mediawiki/y2m.groovy http://evgeny-goldin.org/youtrack/ $*'
alias y2mjb='groovy https://raw.github.com/evgeny-goldin/scripts/master/src/main/groovy/mediawiki/y2m.groovy http://youtrack.jetbrains.net/ $*'


which allows to type

y2m export.csv > 1.txt


Grouping

It is possible to re-order table lines by any field or fields included in the report.

y2m export.csv "Issue Id, Fix versions, Type, State, Summary" "Fix versions, Type, State, Summary" > 1.txt


translates this report into one where issues are sorted by version, type, state, and summary:

Issue Id Fix versions Type State Summary
gc-86 0.5.3.4 Bug Fixed Ant scanner fix - set basedir with canonical path
gc-78 0.5.3.4 Bug Fixed Duplicate log messages
gc-40 0.5.3.4 Feature Fixed FileBean.checksum() - add checksum support
gc-90 0.5.3.4 Feature Fixed FileBean.copyDir() - copy directory recursively
gc-77 0.5.3.4 Feature Fixed FileBean.pack() - add "fullpath" and "prefix" options
gc-85 0.5.3.4 Feature Fixed FileBean.pack() - allow to optionally use TrueZip when packing archives
gc-93 0.5.3.4 Feature Fixed FileBean.pack() - support "filemode" specified as part of include pattern
gc-88 0.5.3.4 Feature Fixed FileBean.pack() - support "fullpath", "prefix" for all archives (zip/tar/tar.gz) and "filemode" for tar/tar.gz
gc-84 0.5.3.4 Feature Fixed FileBean.unpack() - allow to optionally use TrueZip when unpacking archives
gc-87 0.5.3.4 Feature Fixed GeneralBean.executeWithResult() returning String (stdout + stderr)
gc-89 0.5.3.4 Feature Fixed Treat "sar" extension as zip archive
gc-95 0.5.3.4 Feature Fixed VerifyBean.equal() - provide an option whether or not to fail if no files were found
gc-94 0.5.3.4 Feature Fixed VerifyBean.equal() - use combination "directory/pattern" to match files
gc-67 0.5.3.4 Usability Problem Fixed Do not impose logging backend
gc-46 0.5.3.5 Feature Open GCommons.net().http() - HttpBuilder wrapper
gc-76 0.5.3.5 Feature Submitted Make non-required dependencies optional
gc-81 0.5.3.5 Task Submitted Better error message for non-zip files
gc-75 0.5.3.5 Task Submitted Eliminate CodeNarc violations
gc-45 0.5.3.5 Task Submitted Make sure no values are left in recurse() configuration Map after it is read
gc-73 0.5.3.5 Task Submitted Switch to Guice
gc-65 0.5.3.5 Task Submitted svnOp, mvnOp, MW tests


Wiki Syntax Translation

Some YouTrack wiki syntax elements are translated into their MediaWiki analogs. It helps when "Description" field is included into report.

y2m export.csv "Issue Id, Type, Summary, Description" > 1.txt


Issue Id Type Summary Description
gc-90 Feature FileBean.copyDir() - copy directory recursively
GCommons.file().copyDir( File         sourceDirectory,
                         File         destinationDirectory,
                         List<String> includePatterns,
                         List<String> excludePatterns,
                         boolean      failIfNotFound )


Tests are available: 1, 2.
gc-67 Usability Problem Do not impose logging backend For some strange reason GCommons imposes the dev to use Logback. The common approach in a utility lib is to use slf4j and let the client code decide which backend it prefers. More over, if someone uses GCommons and Logback in it's client code, their might be some conflict in logging configuration. Remove that dep please.
gc-40 Feature FileBean.checksum() - add checksum support Test is available.
gc-84 Feature FileBean.unpack() - allow to optionally use TrueZip when unpacking archives See FileBean.unpack() and test.
gc-85 Feature FileBean.pack() - allow to optionally use TrueZip when packing archives
GCommons.file().pack ( File         sourceDirectory,
                       File         destinationArchive,
                       List<String> includes        = null,
                       List<String> excludes        = null,
                       boolean      useTrueZip      = false,
                       ...

Test is availble.
gc-78 Bug Duplicate log messages
gc-77 Feature FileBean.pack() - add "fullpath" and "prefix" options Similarly to Ant <zipfileset>.
Test is available.
gc-88 Feature FileBean.pack() - support "fullpath", "prefix" for all archives (zip/tar/tar.gz) and "filemode" for tar/tar.gz
GCommons.file().pack ( File         sourceDirectory,
                       File         destinationArchive,
                       List<String> includes        = null,
                       List<String> excludes        = null,
                       boolean      useTrueZip      = false,
                       boolean      failIfNotFound  = true,
                       boolean      update          = false,
                       List<String> defaultExcludes = null,
                       String       fullpath        = null,
                       String       prefix          = null )

Tests are available for fullpath/prefix and filemode. Note that to use filemode you need to:

  • Use Ant for packing archives, not TrueZip.
  • Specify filemode as part of include pattern: "*.sh|700" or "*.sh|755"
gc-94 Feature VerifyBean.equal() - use combination "directory/pattern" to match files
GCommons.verify().equal ( File    file1,
                          File    file2,
                          boolean verifyChecksum = true,
                          String  pattern        = null,
                          String  endOfLine      = null,
                          boolean failIfNoFiles  = true )


where "pattern" can be "*.txt" or "*/.txt". If both "file1" and "file2" are directories, then "*.txt" will only check direct children of those directories while "*/.txt" will work recursively, all levels down.

Test is available.
gc-95 Feature VerifyBean.equal() - provide an option whether or not to fail if no files were found
int GCommons.verify().equal( File    file1,
                             File    file2,
                             boolean verifyChecksum = true,
                             String  pattern        = null,
                             String  endOfLine      = null,
                             boolean failIfNoFiles  = true )

Number of files verified is returned. Test is available.
gc-93 Feature FileBean.pack() - support "filemode" specified as part of include pattern
GCommons.file().pack( sourceDirectory, destinationArchive, [ '*.php', '*.sh|755' ], [], false )
GCommons.file().pack( sourceDirectory, destinationArchive, [ '*.php', '*.sh|755' ] )

where "destinationArchive" is a "tar.gz" archive and "useTrueZip" argument is false.
This feature only works when tar.gz archives are packed with Ant.

Test is available.
gc-87 Feature GeneralBean.executeWithResult() returning String (stdout + stderr) GCommons.general().executeWithResult( String command, strategy )
where "strategy" is one of:

  • Commons-Exec
  • Groovy's "command".execute()
  • Java ProcessBuilder

    Test is available.
gc-86 Bug Ant scanner fix - set basedir with canonical path Test is available.
gc-89 Feature Treat "sar" extension as zip archive Test is available.


Lines Numbering

Lines numbering can be added by specifying an additional boolean argument.

y2m export.csv "Issue Id, Type, Summary" "Type, Summary" true > 1.txt


# Issue Id Type Summary
1 gc-86 Bug Ant scanner fix - set basedir with canonical path
2 gc-78 Bug Duplicate log messages
3 gc-40 Feature FileBean.checksum() - add checksum support
4 gc-90 Feature FileBean.copyDir() - copy directory recursively
5 gc-77 Feature FileBean.pack() - add "fullpath" and "prefix" options
6 gc-85 Feature FileBean.pack() - allow to optionally use TrueZip when packing archives
7 gc-93 Feature FileBean.pack() - support "filemode" specified as part of include pattern
8 gc-88 Feature FileBean.pack() - support "fullpath", "prefix" for all archives (zip/tar/tar.gz) and "filemode" for tar/tar.gz
9 gc-84 Feature FileBean.unpack() - allow to optionally use TrueZip when unpacking archives
10 gc-87 Feature GeneralBean.executeWithResult() returning String (stdout + stderr)
11 gc-89 Feature Treat "sar" extension as zip archive
12 gc-95 Feature VerifyBean.equal() - provide an option whether or not to fail if no files were found
13 gc-94 Feature VerifyBean.equal() - use combination "directory/pattern" to match files
14 gc-67 Usability Problem Do not impose logging backend


Examples

Maven Plugins: Fix versions: 0.2.3.1, 0.2.3.4 Subsystem: copy-maven-plugin


y2m export.csv > 1.txt


Issue Id Type Summary
pl-233 Bug <filtering>true</filtering> doesn't filter *.bat file ?
pl-477 Bug Add support for <nonFilteredExtensions>
pl-463 Bug Executable permission bits are not preserved after packing a kit into .tar.gz
pl-470 Bug Fail the build if <includeScope>compile</includeScope> (or any other filtered dependencies) retrieves 0 results
pl-469 Bug Filtered dependency comes from module's "dist" instead of local Maven repo, file name assertion fails
pl-127 Bug If unable to unpack (unknown extension) - doesn't do anything silently
pl-465 Bug Packing non-existing file to tar.gz archive does not fail
pl-473 Feature Add "/**" to directory <include> if pattern is not specified
pl-302 Feature Merge explicit <dependency> and filtered dependencies
pl-434 Feature Provide <targetRoots> property with comma-separated list of target roots
pl-468 Feature Provide an option to avoid copying hidden files
pl-478 Feature Provide an option to control whether TrueZip or Ant is used for packing and unpacking archives
pl-436 Feature Set <dependenciesAtM2> to "true" when possible
pl-466 Feature Support "destFileName", "prefix" for all archives, "filemode" for tar.gz archives
pl-429 Feature Support "prefix" and "destFileName" when archives are updated or created
pl-441 Feature Support <dependency> "destFileName"
pl-442 Feature Support <replace> and <filtering> when files are packed or unpacked
pl-350 Feature Support for <move>true/false</move> when files are copied or archived
pl-451 Feature Switch to TrueZip unpacking by default
pl-467 Feature Treat "sar" extension as zip archive
pl-446 Task If some <include>s have no files matched - do not fail if others had files matched
pl-444 Task When archive is unpacked <process> should contain files unpacked


JetBrains: project: Kotlin Subsystems: {Type checker}


y2mjb export.csv > 1.txt


Issue Id Type Summary
KT-330 Bug Handle mutually recursive definitions more granularly
KT-354 Bug Inner classes are not inherited
KT-328 Bug Local function in function literals cause exceptions
KT-358 Bug Report a warning if a safely accessed variable is known to be non-null by data flow
KT-371 Bug Resolve default parameters for constructors
KT-338 Bug Support autocasts in nested declarations


y2mjb export.csv "Issue Id, Summary, Description" > 1.txt


Issue Id Summary Description
KT-371 Resolve default parameters for constructors
class foo(x : Int = y, y : Int = x) { // None of the references is resolved, no types checked
 
}
KT-358 Report a warning if a safely accessed variable is known to be non-null by data flow
fun notContainsBreak(a: String?, b: String?) {
  if (a != null)
   a?.compareTo("2")
}
KT-354 Inner classes are not inherited
open class Foo() {
  open class Bar() {
 
  }
 
  val b = Bar()
 
  class X() : Bar() {}
}
 
class X() : Foo() {
    class Y() : Bar() {} // ERROR, but shouldn't be
}
KT-338 Support autocasts in nested declarations
if (a is B) {
  class C : X(a) {
    this(i : Int) : this(a) {}
  }
}
KT-330 Handle mutually recursive definitions more granularly
val y = object {
  val a = y;
}
 
val z = y.a;
 
object A {
  val x = A
}
 
val a = object {
  {
    b + 1
  }
  val x = b
  val y = 1
}
 
val b = a.x
val c = a.y // here
 
val x : Int = a.y

The type of y at the marked line could be known, but it appears to be error type
KT-328 Local function in function literals cause exceptions
fun bar() = {
  fun foo() = bar()
  foo()
}


JetBrains: project: TeamCity Subsystem: {Version Control: Git} Priority: Critical State: Fixed


y2mjb export.csv "Issue Id, Type, Priority, State, Summary" "Type, Summary" true > 1.txt


# Issue Id Type Priority State Summary
1 TW-8259 Bug Critical Fixed Cannot select private key when adding a new Git VCS root to build
2 TW-11291 Bug Critical Fixed Confusing "Unsupported version of Git is detected" message on agent
3 TW-9611 Bug Critical Fixed Ensure git repository communications have timeout
4 TW-13009 Bug Critical Fixed Exception on git submodule update on windows
5 TW-9574 Bug Critical Fixed Git : Build hangs forever during stage called "Checking for changes"
6 TW-12143 Bug Critical Fixed Git checking for changes threads can hang (patched jgit 0.7.1 case)
7 TW-14606 Bug Critical Fixed Git should not start several fetch processes for collect/fetch changes from same repository
8 TW-10489 Bug Critical Fixed Long builds Git checking for changes stage
9 TW-7949 Bug Critical Fixed NPE on manual labeling
10 TW-10401 Bug Critical Fixed Occasional TransportException: Remote does not have refs/heads/master available for fetch.
11 TW-8979 Bug Critical Fixed Teamcity, when building a merge commit, is only including changes from one side of the merge
12 TW-13330 Bug Critical Fixed git fetch fails with unspecific error
13 TW-10741 Feature Critical Fixed Agent-side checkout for Git
14 TW-8056 Feature Critical Fixed Remote Run for Git in IDEA
15 TW-14104 Performance Problem Critical Fixed "Searching for suitable configurations" is too slow
Personal tools