MediaWiki Tools
From Evgeny Goldin
While working with this Wiki I developed a number of Groovy scripts to keep all pages organized. Those scripts are not versioned, they work for me but they may not work for you - we may have different needs and input files.
Still, I'd love to hear whenever it happens.
Contents |
"Links" - Verifies MediaWiki links
links.groovy verifies all internal and cross-page links in MediaWiki files are correct.
- Internal links are those of form
[[#Usage|link text]]where"Usage"is an existing section in the same page. - Cross-page links are those of form
"http://wiki/MyPage#Section"where"Section"is an existing section in the referenced"MyPage".
Usage
groovy links.groovy <directory> [files include pattern]
- If no include pattern is provided then all files in directory specified are assumed to be MediaWiki pages.
- Include pattern is the one recognized by Ant's
DirectoryScanner. - External links are links starting with
"http"and having a"#"as part of the URL. For"http://wiki/MyPage#Section"the link is verified only if"MyPage.anything"file was found when scanning for files. - Execution fails if broken links are found.
Example execution (taken from this test verifying this Wiki links):
$ groovy links.groovy /Users/evgenyg/Projects/wiki/ '**/*.txt' ... [/Users/evgenyg/Projects/wiki/maven-plugins/assert-maven-plugin.txt] - 0 internal links [/Users/evgenyg/Projects/wiki/maven-plugins/copy-maven-plugin.txt] - 47 internal links [/Users/evgenyg/Projects/wiki/maven-plugins/find-maven-plugin.txt] - 0 internal links [/Users/evgenyg/Projects/wiki/maven-plugins/jenkis-maven-plugin.txt] - 4 internal links ... [/Users/evgenyg/Projects/wiki/gcommons/Main.txt] - 116 external links [/Users/evgenyg/Projects/wiki/gradle-plugins/Gradle-CodeNarc-plugin.txt] - 15 external links [/Users/evgenyg/Projects/wiki/gradle-plugins/Gradle-about-plugin.txt] - 16 external links [/Users/evgenyg/Projects/wiki/gradle-plugins/Gradle-duplicates-plugin.txt] - 15 external links [/Users/evgenyg/Projects/wiki/gradle-plugins/Main.txt] - 9 external links [/Users/evgenyg/Projects/wiki/maven-plugins/Duplicates-finder-plugin.txt] - 17 external links [/Users/evgenyg/Projects/wiki/maven-plugins/Main.txt] - 194 external links ... [39] files processed (486 ms)
"Spaces" - Fixes MeddiaWiki pages spacing
spaces.groovy updates MediaWiki files so that every "= Section =" is preceded by two empty lines at least. It also trims all trailing spaces in each line.
Usage
groovy spaces.groovy <directory> [files include pattern] [fail if update]
- If no include pattern is provided then all files in directory specified are assumed to be MediaWiki pages.
- Include pattern is the one recognized by Ant's
DirectoryScanner. - If last argument is
truethen execution fails whenever any page was updated. It helps to verify all pages have the correct spacing set.
Example execution (taken from this test verifying this Wiki pages):
$ groovy spaces.groovy /Users/evgenyg/Projects/wiki/ '**/*.txt' Fixing spaces in [/Users/evgenyg/Projects/wiki]/[**/*.txt], fail if update [false] [/Users/evgenyg/Projects/wiki/MediaWiki Tools/Main.txt] updated Fixing spaces in [/Users/evgenyg/Projects/wiki]/[**/*.txt], fail if update [false] - Done (587 ms) $ groovy spaces.groovy /Users/evgenyg/Projects/wiki/ '**/*.txt' true Fixing spaces in [/Users/evgenyg/Projects/wiki]/[**/*.txt], fail if update [true] Fixing spaces in [/Users/evgenyg/Projects/wiki]/[**/*.txt], fail if update [true] - Done (593 ms)
"g2m" - Converts GitHub flavored Markdown pages to MediaWiki
g2m.groovy converts GitHub flavored Markdown files to MediaWiki syntax. It helps to convert "README.markdown" files and make them as close as possible to be used in MediaWiki.
For example, Spock extensions page in this Wiki is a converted version of the corresponding "README.markdown" with minor manual edits applied.
Usage
groovy g2m.groovy <markdown file> <output file>
"y2m" - Converts YouTrack CSV export files to MediaWiki
y2m.groovy translates YouTrack CSV exports into MediaWiki tables.
| Issue Id | Type | Summary |
|---|---|---|
| gc-86 | Bug | Ant scanner fix - set basedir with canonical path |
| gc-78 | Bug | Duplicate log messages |
| gc-40 | Feature | FileBean.checksum() - add checksum support |
| gc-90 | Feature | FileBean.copyDir() - copy directory recursively |
| gc-77 | Feature | FileBean.pack() - add "fullpath" and "prefix" options |
| gc-85 | Feature | FileBean.pack() - allow to optionally use TrueZip when packing archives |
| gc-93 | Feature | FileBean.pack() - support "filemode" specified as part of include pattern |
| gc-88 | Feature | FileBean.pack() - support "fullpath", "prefix" for all archives (zip/tar/tar.gz) and "filemode" for tar/tar.gz |
| gc-84 | Feature | FileBean.unpack() - allow to optionally use TrueZip when unpacking archives |
| gc-87 | Feature | GeneralBean.executeWithResult() returning String (stdout + stderr) |
| gc-89 | Feature | Treat "sar" extension as zip archive |
| gc-95 | Feature | VerifyBean.equal() - provide an option whether or not to fail if no files were found |
| gc-94 | Feature | VerifyBean.equal() - use combination "directory/pattern" to match files |
| gc-67 | Usability Problem | Do not impose logging backend |
Usage
If you run the script without any arguments it displays the usage:
$ groovy y2m.groovy ------------------------------------------------------------------------------------------------------------------------------ Usage: groovy y2m.groovy <YouTrack URL> <CSV file> [<Fields>] [<Group-By-Fields>] [<Add #>] ------------------------------------------------------------------------------------------------------------------------------ YouTrack URL - Base URL of YouTrack application, like "http://youtrack.jetbrains.net" or "http://evgeny-goldin.org/youtrack" CSV file - "Issues in CSV"-exported file from YouTrack Fields - (optional) comma-separated list of fields to use in MediaWiki table, "${ defaultFields.join( ', ' )}" by default Group-By-Fields - (optional) comma-separated list of fields to group table rows by, "${ defaultGroupByFields.join( ', ' )}" for the default fields Add # - (optional) true/false, whether to add a '#' column with a running counter for each issue, false by default If -Dy2mFile=file is specifed then result is written to this file instead of writing it to stdout. ------------------------------------------------------------------------------------------------------------------------------
The table above was generated by running:
groovy y2m.groovy http://evgeny-goldin.org/youtrack export.csv > 1.txt
Since Groovy 1.8.3 allows to run scripts retrieved over HTTP I have the following alias defined in my "~/.bash_profile":
alias y2m='groovy https://raw.github.com/evgeny-goldin/scripts/master/src/main/groovy/mediawiki/y2m.groovy http://evgeny-goldin.org/youtrack/ $*' alias y2mjb='groovy https://raw.github.com/evgeny-goldin/scripts/master/src/main/groovy/mediawiki/y2m.groovy http://youtrack.jetbrains.net/ $*'
which allows to type
y2m export.csv > 1.txt
Grouping
It is possible to re-order table lines by any field or fields included in the report.
y2m export.csv "Issue Id, Fix versions, Type, State, Summary" "Fix versions, Type, State, Summary" > 1.txt
translates this report into one where issues are sorted by version, type, state, and summary:
| Issue Id | Fix versions | Type | State | Summary |
|---|---|---|---|---|
| gc-86 | 0.5.3.4 | Bug | Fixed | Ant scanner fix - set basedir with canonical path |
| gc-78 | 0.5.3.4 | Bug | Fixed | Duplicate log messages |
| gc-40 | 0.5.3.4 | Feature | Fixed | FileBean.checksum() - add checksum support |
| gc-90 | 0.5.3.4 | Feature | Fixed | FileBean.copyDir() - copy directory recursively |
| gc-77 | 0.5.3.4 | Feature | Fixed | FileBean.pack() - add "fullpath" and "prefix" options |
| gc-85 | 0.5.3.4 | Feature | Fixed | FileBean.pack() - allow to optionally use TrueZip when packing archives |
| gc-93 | 0.5.3.4 | Feature | Fixed | FileBean.pack() - support "filemode" specified as part of include pattern |
| gc-88 | 0.5.3.4 | Feature | Fixed | FileBean.pack() - support "fullpath", "prefix" for all archives (zip/tar/tar.gz) and "filemode" for tar/tar.gz |
| gc-84 | 0.5.3.4 | Feature | Fixed | FileBean.unpack() - allow to optionally use TrueZip when unpacking archives |
| gc-87 | 0.5.3.4 | Feature | Fixed | GeneralBean.executeWithResult() returning String (stdout + stderr) |
| gc-89 | 0.5.3.4 | Feature | Fixed | Treat "sar" extension as zip archive |
| gc-95 | 0.5.3.4 | Feature | Fixed | VerifyBean.equal() - provide an option whether or not to fail if no files were found |
| gc-94 | 0.5.3.4 | Feature | Fixed | VerifyBean.equal() - use combination "directory/pattern" to match files |
| gc-67 | 0.5.3.4 | Usability Problem | Fixed | Do not impose logging backend |
| gc-46 | 0.5.3.5 | Feature | Open | GCommons.net().http() - HttpBuilder wrapper |
| gc-76 | 0.5.3.5 | Feature | Submitted | Make non-required dependencies optional |
| gc-79 | 0.5.3.5 | Task | Submitted | "stars" method update |
| gc-81 | 0.5.3.5 | Task | Submitted | Better error message for non-zip files |
| gc-75 | 0.5.3.5 | Task | Submitted | Eliminate CodeNarc violations |
| gc-45 | 0.5.3.5 | Task | Submitted | Make sure no values are left in recurse() configuration Map after it is read |
| gc-73 | 0.5.3.5 | Task | Submitted | Switch to Guice |
| gc-65 | 0.5.3.5 | Task | Submitted | svnOp, mvnOp, MW tests |
Wiki Syntax Translation
Some YouTrack wiki syntax elements are translated into their MediaWiki analogs. It helps when "Description" field is included into report.
y2m export.csv "Issue Id, Type, Summary, Description" > 1.txt
| Issue Id | Type | Summary | Description |
|---|---|---|---|
| gc-90 | Feature | FileBean.copyDir() - copy directory recursively |
GCommons.file().copyDir( File sourceDirectory,
File destinationDirectory,
List<String> includePatterns,
List<String> excludePatterns,
boolean failIfNotFound )Tests are available: 1, 2. |
| gc-67 | Usability Problem | Do not impose logging backend | For some strange reason GCommons imposes the dev to use Logback. The common approach in a utility lib is to use slf4j and let the client code decide which backend it prefers. More over, if someone uses GCommons and Logback in it's client code, their might be some conflict in logging configuration. Remove that dep please. |
| gc-40 | Feature | FileBean.checksum() - add checksum support | Test is available. |
| gc-84 | Feature | FileBean.unpack() - allow to optionally use TrueZip when unpacking archives | See FileBean.unpack() and test. |
| gc-85 | Feature | FileBean.pack() - allow to optionally use TrueZip when packing archives |
GCommons.file().pack ( File sourceDirectory,
File destinationArchive,
List<String> includes = null,
List<String> excludes = null,
boolean useTrueZip = false,
...Test is availble. |
| gc-78 | Bug | Duplicate log messages | |
| gc-77 | Feature | FileBean.pack() - add "fullpath" and "prefix" options | Similarly to Ant <zipfileset>.Test is available. |
| gc-88 | Feature | FileBean.pack() - support "fullpath", "prefix" for all archives (zip/tar/tar.gz) and "filemode" for tar/tar.gz |
GCommons.file().pack ( File sourceDirectory,
File destinationArchive,
List<String> includes = null,
List<String> excludes = null,
boolean useTrueZip = false,
boolean failIfNotFound = true,
boolean update = false,
List<String> defaultExcludes = null,
String fullpath = null,
String prefix = null )Tests are available for fullpath/prefix and filemode. Note that to use filemode you need to:
|
| gc-94 | Feature | VerifyBean.equal() - use combination "directory/pattern" to match files |
GCommons.verify().equal ( File file1,
File file2,
boolean verifyChecksum = true,
String pattern = null,
String endOfLine = null,
boolean failIfNoFiles = true )where "pattern" can be "*.txt" or "*/.txt". If both "file1" and "file2" are directories, then "*.txt" will only check direct children of those directories while "*/.txt" will work recursively, all levels down.Test is available. |
| gc-95 | Feature | VerifyBean.equal() - provide an option whether or not to fail if no files were found |
int GCommons.verify().equal( File file1,
File file2,
boolean verifyChecksum = true,
String pattern = null,
String endOfLine = null,
boolean failIfNoFiles = true )Number of files verified is returned. Test is available. |
| gc-93 | Feature | FileBean.pack() - support "filemode" specified as part of include pattern |
GCommons.file().pack( sourceDirectory, destinationArchive, [ '*.php', '*.sh|755' ], [], false ) GCommons.file().pack( sourceDirectory, destinationArchive, [ '*.php', '*.sh|755' ] ) where "destinationArchive" is a "tar.gz" archive and "useTrueZip" argument is false.This feature only works when tar.gz archives are packed with Ant.Test is available. |
| gc-87 | Feature | GeneralBean.executeWithResult() returning String (stdout + stderr) | GCommons.general().executeWithResult( String command, strategy )where "strategy" is one of:
|
| gc-86 | Bug | Ant scanner fix - set basedir with canonical path | Test is available. |
| gc-89 | Feature | Treat "sar" extension as zip archive | Test is available. |
Lines Numbering
Lines numbering can be added by specifying an additional boolean argument.
y2m export.csv "Issue Id, Type, Summary" "Type, Summary" true > 1.txt
| # | Issue Id | Type | Summary |
|---|---|---|---|
| 1 | gc-86 | Bug | Ant scanner fix - set basedir with canonical path |
| 2 | gc-78 | Bug | Duplicate log messages |
| 3 | gc-40 | Feature | FileBean.checksum() - add checksum support |
| 4 | gc-90 | Feature | FileBean.copyDir() - copy directory recursively |
| 5 | gc-77 | Feature | FileBean.pack() - add "fullpath" and "prefix" options |
| 6 | gc-85 | Feature | FileBean.pack() - allow to optionally use TrueZip when packing archives |
| 7 | gc-93 | Feature | FileBean.pack() - support "filemode" specified as part of include pattern |
| 8 | gc-88 | Feature | FileBean.pack() - support "fullpath", "prefix" for all archives (zip/tar/tar.gz) and "filemode" for tar/tar.gz |
| 9 | gc-84 | Feature | FileBean.unpack() - allow to optionally use TrueZip when unpacking archives |
| 10 | gc-87 | Feature | GeneralBean.executeWithResult() returning String (stdout + stderr) |
| 11 | gc-89 | Feature | Treat "sar" extension as zip archive |
| 12 | gc-95 | Feature | VerifyBean.equal() - provide an option whether or not to fail if no files were found |
| 13 | gc-94 | Feature | VerifyBean.equal() - use combination "directory/pattern" to match files |
| 14 | gc-67 | Usability Problem | Do not impose logging backend |
Examples
Maven Plugins: Fix versions: 0.2.3.1, 0.2.3.4 Subsystem: copy-maven-plugin
y2m export.csv > 1.txt
| Issue Id | Type | Summary |
|---|---|---|
| pl-233 | Bug | <filtering>true</filtering> doesn't filter *.bat file ? |
| pl-477 | Bug | Add support for <nonFilteredExtensions> |
| pl-463 | Bug | Executable permission bits are not preserved after packing a kit into .tar.gz |
| pl-470 | Bug | Fail the build if <includeScope>compile</includeScope> (or any other filtered dependencies) retrieves 0 results |
| pl-469 | Bug | Filtered dependency comes from module's "dist" instead of local Maven repo, file name assertion fails |
| pl-127 | Bug | If unable to unpack (unknown extension) - doesn't do anything silently |
| pl-465 | Bug | Packing non-existing file to tar.gz archive does not fail |
| pl-473 | Feature | Add "/**" to directory <include> if pattern is not specified |
| pl-302 | Feature | Merge explicit <dependency> and filtered dependencies |
| pl-434 | Feature | Provide <targetRoots> property with comma-separated list of target roots |
| pl-468 | Feature | Provide an option to avoid copying hidden files |
| pl-478 | Feature | Provide an option to control whether TrueZip or Ant is used for packing and unpacking archives |
| pl-436 | Feature | Set <dependenciesAtM2> to "true" when possible |
| pl-466 | Feature | Support "destFileName", "prefix" for all archives, "filemode" for tar.gz archives |
| pl-429 | Feature | Support "prefix" and "destFileName" when archives are updated or created |
| pl-441 | Feature | Support <dependency> "destFileName" |
| pl-442 | Feature | Support <replace> and <filtering> when files are packed or unpacked |
| pl-350 | Feature | Support for <move>true/false</move> when files are copied or archived |
| pl-451 | Feature | Switch to TrueZip unpacking by default |
| pl-467 | Feature | Treat "sar" extension as zip archive |
| pl-446 | Task | If some <include>s have no files matched - do not fail if others had files matched |
| pl-444 | Task | When archive is unpacked <process> should contain files unpacked |
JetBrains: project: Kotlin Subsystems: {Type checker}
y2mjb export.csv > 1.txt
| Issue Id | Type | Summary |
|---|---|---|
| KT-330 | Bug | Handle mutually recursive definitions more granularly |
| KT-354 | Bug | Inner classes are not inherited |
| KT-328 | Bug | Local function in function literals cause exceptions |
| KT-358 | Bug | Report a warning if a safely accessed variable is known to be non-null by data flow |
| KT-371 | Bug | Resolve default parameters for constructors |
| KT-338 | Bug | Support autocasts in nested declarations |
y2mjb export.csv "Issue Id, Summary, Description" > 1.txt
| Issue Id | Summary | Description |
|---|---|---|
| KT-371 | Resolve default parameters for constructors |
class foo(x : Int = y, y : Int = x) { // None of the references is resolved, no types checked
} |
| KT-358 | Report a warning if a safely accessed variable is known to be non-null by data flow |
fun notContainsBreak(a: String?, b: String?) {
if (a != null)
a?.compareTo("2")
} |
| KT-354 | Inner classes are not inherited |
open class Foo() {
open class Bar() {
}
val b = Bar()
class X() : Bar() {}
}
class X() : Foo() {
class Y() : Bar() {} // ERROR, but shouldn't be
} |
| KT-338 | Support autocasts in nested declarations |
if (a is B) {
class C : X(a) {
this(i : Int) : this(a) {}
}
} |
| KT-330 | Handle mutually recursive definitions more granularly |
val y = object {
val a = y;
}
val z = y.a;
object A {
val x = A
}
val a = object {
{
b + 1
}
val x = b
val y = 1
}
val b = a.x
val c = a.y // here
val x : Int = a.yThe type of y at the marked line could be known, but it appears to be error type
|
| KT-328 | Local function in function literals cause exceptions |
fun bar() = {
fun foo() = bar()
foo()
} |
JetBrains: project: TeamCity Subsystem: {Version Control: Git} Priority: Critical State: Fixed
y2mjb export.csv "Issue Id, Type, Priority, State, Summary" "Type, Summary" true > 1.txt
| # | Issue Id | Type | Priority | State | Summary |
|---|---|---|---|---|---|
| 1 | TW-8259 | Bug | Critical | Fixed | Cannot select private key when adding a new Git VCS root to build |
| 2 | TW-11291 | Bug | Critical | Fixed | Confusing "Unsupported version of Git is detected" message on agent |
| 3 | TW-9611 | Bug | Critical | Fixed | Ensure git repository communications have timeout |
| 4 | TW-13009 | Bug | Critical | Fixed | Exception on git submodule update on windows |
| 5 | TW-9574 | Bug | Critical | Fixed | Git : Build hangs forever during stage called "Checking for changes" |
| 6 | TW-12143 | Bug | Critical | Fixed | Git checking for changes threads can hang (patched jgit 0.7.1 case) |
| 7 | TW-14606 | Bug | Critical | Fixed | Git should not start several fetch processes for collect/fetch changes from same repository |
| 8 | TW-10489 | Bug | Critical | Fixed | Long builds Git checking for changes stage |
| 9 | TW-7949 | Bug | Critical | Fixed | NPE on manual labeling |
| 10 | TW-10401 | Bug | Critical | Fixed | Occasional TransportException: Remote does not have refs/heads/master available for fetch. |
| 11 | TW-8979 | Bug | Critical | Fixed | Teamcity, when building a merge commit, is only including changes from one side of the merge |
| 12 | TW-13330 | Bug | Critical | Fixed | git fetch fails with unspecific error |
| 13 | TW-10741 | Feature | Critical | Fixed | Agent-side checkout for Git |
| 14 | TW-8056 | Feature | Critical | Fixed | Remote Run for Git in IDEA |
| 15 | TW-14104 | Performance Problem | Critical | Fixed | "Searching for suitable configurations" is too slow |
