During the development of WordPress 3.4 I have spent some time working on improving the rewrite API. One of the tickets this involved was #16303: “Improve documentation and usability of WP_Rewrite Endpoint support”. Endpoints are a really cool feature of the rewrite API, but unfortunately also little known and misunderstood. So, with this post my aim is to get more plugin developers to read and understand the new and improved endpoint documentation.
What are endpoints?
Using endpoints allows you to easily create rewrite rules to catch the normal WordPress URLs, but with a little extra at the end. For example, you could use an endpoint to match all post URLs followed by “gallery” and display all of the images used in a post, e.g. http://example.com/my-fantastic-post/gallery/.
A simple case like this is relatively easy to achieve with your own custom rewrite rules. However, the power of endpoints shines for more complex situations. What if you wanted to recognise URLs for posts and pages ending with “gallery”? What if you wanted to be able to catch multiple different archive URLs, e.g. day, month, year and category archives, with “xml” appended in order to output an XML representation of the archive? For these situations endpoints are very useful as they allow you to add a string to the end of multiple rewrite structures with a single function call.
How to use them
There is one function for interacting with endpoints: add_rewrite_endpoint(). It takes two parameters $name and $places.
$name is a string and is, wait for it… the name of the endpoint. $name is what is used in the URL and is the name of the query variable that the endpoint URL will be rewritten to. For example, an endpoint named “print” added to post permalinks would use a URL like http://example.com/my-awesome-post/print/.
$places is an integer value which represents the locations (places) to which the endpoint will be added, e.g. posts, pages or year achives. To understand $places you need to learn about the endpoint mask constants.
In wp-includes/rewrite.php (browse wp-includes/rewrite.php on Trac) a number of constants are defined all with names beginning with “EP_”:
define('EP_NONE', 0); // 0000000000000
define('EP_PERMALINK', 1); // 0000000000001
define('EP_ATTACHMENT', 2); // 0000000000010
define('EP_DATE', 4); // 0000000000100
define('EP_YEAR', 8); // 0000000001000
// ...
define('EP_PAGES', 4096); // 1000000000000
define('EP_ALL', 8191); // 1111111111111
These are the endpoint masks which describe sets of URLs; post permalinks are described by EP_PERMALINK, year archives are EP_YEAR, etc. They should be thought of in terms of their binary values (see the comment I’ve added to the end of each line). Every EP_* mask, except for EP_ALL, is a different power of two and so has a different bit set to one. This allows us to build up combinations of endpoint masks by using the bitwise OR operator:
// all posts or attachments
EP_PERMALINK | EP_ATTACHMENT // 0000000000011
// all full dates (yyyy/mm/dd), years or pages
EP_DATE | EP_YEAR | EP_PAGES // 1000000001100
$places should also be thought of as a binary number. It should be set to one of the EP_* constants or a combination of them using the bitwise OR operator. If we wanted to add our endpoint to all post permalinks we would use EP_PERMALINK. For both posts and pages: EP_PERMALINK | EP_PAGES. For posts, pages, and categories: EP_PERMALINK | EP_PAGES | EP_CATEGORIES. There is also a special value to add an endpoint to all URLs that support endpoints: EP_ALL.
NB: The values of the EP_* constants are not guaranteed to stay the same which is why you must say $places = EP_PERMALINK and not $places = 2. This is particularly important for EP_ALL which will change every time a new endpoint mask is added.
It’s time to put this information into practise. The running example will be a plugin that adds JSON representations of our content using a new rewrite endpoint called “json”. So, the goal is to get URLs such as http://example.com/about/json/ to return a JSON response that gives information about the “about” page. To add the “json” endpoint to post and page rewrite structures:
add_rewrite_endpoint( 'json', EP_PERMALINK | EP_PAGES );
This is called in a function hooked into the init action:
function makeplugins_add_json_endpoint() {
add_rewrite_endpoint( 'json', EP_PERMALINK | EP_PAGES );
}
add_action( 'init', 'makeplugins_add_json_endpoint' );
Now we want to act on requests for JSON content. This is done by hooking into template_redirect. We want to detect appropriate requests and include our custom template for serving up posts and pages in JSON format:
function makeplugins_json_template_redirect() {
global $wp_query;
// if this is not a request for json or a singular object then bail
if ( ! isset( $wp_query->query_vars['json'] ) || ! is_singular() )
return;
// include custom template
include dirname( __FILE__ ) . '/json-template.php';
exit;
}
add_action( 'template_redirect', 'makeplugins_json_template_redirect' );
And we’re done. For a full example plugin see https://gist.github.com/2891111.
How do they work?
The best way to understand how anything works is to take a look at the source, so let’s do that. Endpoints are added with the add_rewrite_endpoint() function in wp-includes/rewrite.php:
/**
* Add an endpoint, like /trackback/.
*
* Adding an endpoint creates extra rewrite rules for each of the matching
* places specified by the provided bitmask. For example:
*
* <code>
* add_rewrite_endpoint( 'json', EP_PERMALINK | EP_PAGES );
* </code>
*
* will add a new rewrite rule ending with "json(/(.*))?/?$" for every permastruct
* that describes a permalink (post) or page. This is rewritten to "json=$match"
* where $match is the part of the URL matched by the endpoint regex (e.g. "foo" in
* "/json/foo/").
*
* A new query var with the same name as the endpoint will also be created.
*
* When specifying $places ensure that you are using the EP_* constants (or a
* combination of them using the bitwise OR operator) as their values are not
* guaranteed to remain static (especially EP_ALL).
*
* Be sure to flush the rewrite rules - flush_rewrite_rules() - when your plugin gets
* activated and deactivated.
*
* @since 2.1.0
* @see WP_Rewrite::add_endpoint()
* @global object $wp_rewrite
*
* @param string $name Name of the endpoint.
* @param int $places Endpoint mask describing the places the endpoint should be added.
*/
function add_rewrite_endpoint( $name, $places ) {
global $wp_rewrite;
$wp_rewrite->add_endpoint( $name, $places );
}
So this is just a wrapper for the add_endpoint() method of the WP_Rewrite class. Although the (excellent!) documentation gives us some clues as to what it does we’ll have to dig deeper to find the how:
/**
* Add an endpoint, like /trackback/.
*
* See {@link add_rewrite_endpoint()} for full documentation.
*
* @see add_rewrite_endpoint()
* @since 2.1.0
* @access public
* @uses WP::add_query_var()
*
* @param string $name Name of the endpoint.
* @param int $places Endpoint mask describing the places the endpoint should be added.
*/
function add_endpoint($name, $places) {
global $wp;
$this->endpoints[] = array ( $places, $name );
$wp->add_query_var($name);
}
Another very short and simple function. All it does is append the two parameters passed to it to the private $endpoints property of the WP_Rewrite class and also add a new query variable using WP::add_query_var().
Okay, so that’s still not useful for a full understanding of endpoints. All we know is that the arguments you pass to add_rewrite_endpoint() are stored in a private array of the $wp_rewrite global. To find out more we’ll have to search wp-includes/rewrite.php for “>endpoints” (i.e. code accessing the WP_Rewrite::$endpoints property). There are only three references to this: WP_Rewrite::add_endpoint() we have seen, WP_Rewrite::init() is boring (initialising the array), and the third is WP_Rewrite::generate_rewrite_rules():
$ep_query_append = array ();
foreach ( (array) $this->endpoints as $endpoint) {
//match everything after the endpoint name, but allow for nothing to appear there
$epmatch = $endpoint[1] . '(/(.*))?/?$';
//this will be appended on to the rest of the query for each dir
$epquery = '&' . $endpoint[1] . '=';
$ep_query_append[$epmatch] = array ( $endpoint[0], $epquery );
}
// ... a lot of code removed ...
foreach ( (array) $ep_query_append as $regex => $ep) {
//add the endpoints on if the mask fits
if ( $ep[0] & $ep_mask || $ep[0] & $ep_mask_specific )
$rewrite[$match . $regex] = $index . '?' . $query . $ep[1] . $this->preg_index($num_toks + 2);
}
In the code above the first foreach is looping through the defined endpoints and building a new array called $ep_query_append. This new array uses regular expressions that match a specific endpoint as keys and the values are the endpoint $places and $epquery which is a partial query string to append to a full query. So, for our JSON endpoint example we would get:
$ep_query_append[ 'json(/(.*))?/?$' ] = array( EP_PERMALINK | EP_PAGES, '&json=' );
The second loop generates the final rewrite rules for our endpoint. It loops through $ep_query_append checking if the current permastructure being generated has an endpoint mask, $ep_mask, that matches any of the endpoints. If the bitwise AND produces a non-zero value then there’s a match and the endpoint rewrite rules should be added to this permastructure.
For our JSON example, if WP_Rewrite::generate_rewrites_rules() has been called for the posts permalink structure then $ep_mask = EP_PERMALINK and $ep[0] = EP_PERMALINK | EP_PAGES. The bitwise AND of these values produces 1, therefore a new entry is added to $rewrite. Assuming that the post permalink structure is “/%postname%/” it would look something like:
$rewrite[ '([^/]+)/json(/(.*))?/?$' ] = 'index.php?name=$1&json=$3'
This is the final rewrite rule for our JSON endpoint applied to post permalinks. It matches a request for “post-slug/json/” and sets up the appropriate query variables “name” and “json”. Our template_redirect hook now picks this up and produces the required response.
And you made it to the end, phew! Time for a drink…
Conclusion
I hope that after all of that you understand how to use endpoints and how they work. If you have any questions please don’t hesitate to ask them in the comments. Always remember that the best way to understand a function is to look at the source and follow its execution.
silentshin, 
sweetee, 
anantpathak, 
hawkins.barry are discussing. 
Pippin Williamson 4:48 pm on September 13, 2012 Permalink |
Excellent!
logikal16 5:00 pm on September 13, 2012 Permalink |
That’s great news, thanks!
Brad Touesnard 5:03 pm on September 13, 2012 Permalink |
Nice!
Chuck Reynolds 5:04 pm on September 13, 2012 Permalink |
This is great. Shouldn’t we also do the same for readme files? With the recent security talks concerning the version number being in the readme’s in each plugin dir.
Otto 5:15 pm on September 13, 2012 Permalink |
I don’t know what talks you speak of in particular, but a) no and b) knowing the version of code being used is not an inherent security risk.
A security risk is caused by running insecure code. The solution is not to attempt to hide the version, but to update the code to eliminate the security risk. This sort of thinking is along the same lines of hiding the WordPress version number from public view. Hiding your version number doesn’t improve security in the slightest. Fixing security related issues does.
Chuck Reynolds 10:46 pm on September 13, 2012 Permalink |
true, i can’t find the specific thing I read the other day but… whatever. just thinking out loud.
Moved all my plugins’ screenshots to /assets/ from trunk and latest tag.
Ian Dunn 7:54 pm on September 14, 2012 Permalink |
I think that’s only half true. Obviously, fixing security bugs is the most important thing, but security-through-obscurity isn’t inherently bad, if you understand it correctly.
Security shouldn’t be thought of as a binary thing — you’re either “secure” of “insecure” — and there are no silver bullets. Instead, it should be thought of in layers. Security-through-obscurity is a good first layer. It keeps the script kiddies away, so that you can focus your time on defending against real attackers.
It’d be stupid to rely on obscurity as your only layer, but it’s a good practice to employ it as the first layer among many.
Ben Lobaugh (blobaugh) 5:12 pm on September 13, 2012 Permalink |
Thanks Otto
@Chuck – good point, you would lose the tagged readme files though. Maybe the readme.txt file should just be blocked from serving?
scribu 5:53 pm on September 13, 2012 Permalink |
This makes sense. I just moved the screenshots over for all of my plugins.
I noticed that the readme.txt example doesn’t point to the new
/assets/dir yet.Otto 6:06 pm on September 13, 2012 Permalink |
Didn’t think of that one. Fixed.
Jon Brown 8:52 pm on September 13, 2012 Permalink |
http://wordpress.org/extend/plugins/about/faq/ needs updating too.
Thomas_Michaela 6:40 pm on September 13, 2012 Permalink |
Is there any discussion / ideas how to redesign the “popular plugins” section, so authors don’t need to fake their download numbers anymore?
http://wordpress.org/support/topic/plugin-all-in-one-seo-pack-another-update-with-no-changes-this-plugin-is-just-scam?replies=8
Otto 7:18 pm on September 13, 2012 Permalink |
That doesn’t appear to be faked to me, just a bad development practice.
Edit: Note that if you have hard evidence other than a one-time-thing, then that’s different. I’ll burn and salt the earth with the ashes if you can find a real case of somebody gaming our systems, but this does not appear to be such a case.
Jon Brown 7:36 pm on September 13, 2012 Permalink |
This all sounds great, nice work!
Since I’m still a little slow when it comes to SVN though… Am I correct that /Assets is outside of version control? If that’s that case, I think that alone is reason to keep the readme files inside trunk, but even if that is not the case I’d still prefer seeing readme files in the zips.
Otto 7:45 pm on September 13, 2012 Permalink |
No, you’re not correct, and I’m not sure I understand the question.
SVN is a version control system. Everything in it is “in version control”. So… yeah, you’ve left me confused.
And there’s no plan (nor reason) to put readme.txt files anywhere that isn’t part of the plugin and isn’t in the ZIP file. That’s not under discussion nor up for debate, really. Readme’s should be sent with the plugin files.
Jon Brown 7:52 pm on September 13, 2012 Permalink |
Thanks Otto, I poked around trac and figured it out…
I was thinking that the /Assets folder was outside of version control because it wasn’t inside /branches or /trunk, it’s clear that’s not accurate since it gets assigned revision numbers. The svn tree /branch /trunk /tags just always confuses me…. someday it’ll click.
Otto 7:55 pm on September 13, 2012 Permalink |
Try the writeup I wrote about it here:
http://make.wordpress.org/plugins/2012/06/09/the-plugins-directory-and-readme-txt-files/
Edward Caissie 8:03 pm on September 13, 2012 Permalink |
I must be slow or its not enough coffee today … to implement this new ‘assets’ directory the plugin author needs to send a commit with it included, correct? In other words, the assets directory is not being (has not been?) populated into the existing repository plugins (nor is it part of new plugin repositories).
Otto 8:13 pm on September 13, 2012 Permalink |
Correct, the /assets directory is not created for you. You can create it at the root of your plugin directory, alongside the trunk and tags and branches directories, and use “svn add” to add it and any files inside it to your next commit.
Edward Caissie 8:29 pm on September 13, 2012 Permalink |
Thanks. I had tested that with an existing plugin as well as checking out a newly created plugin repository as well. Just been having SVN client issues recently and wanted to make sure I was not _doing_it_wrong().
Matt Mullenweg 6:46 pm on September 14, 2012 Permalink |
Can we add it to the script that sets up directories for new plugins?
Otto 6:50 pm on September 14, 2012 Permalink |
We could, just haven’t seen the need to do so. Not everybody needs it, in the same way that uber-simple plugins don’t really need a readme.txt file.
I’ll go ahead and add it though.
elfin 8:58 pm on September 13, 2012 Permalink |
excellent – one reason why I never included any to begin with was because of the file size issue. Nice work again Otto.
shazdeh 10:38 pm on September 13, 2012 Permalink |
Totally makes sense!
Greenweb 1:30 pm on September 14, 2012 Permalink |
Great update to the system !
XYZScripts 6:08 am on September 15, 2012 Permalink |
Good move. Really saves on zip size.
Next we should separate out readme.txt from folder as Chuck said.
I think right now readme.txt is parsed from trunk until stable tag is found and then it starts parsing stable tag readme.
So we have put this file in both trunk and tag which is simply duplicate.
I dont think readme.txt is needed in s wordpress installation as it reads info from plugin main file (Correct me if an wrong)
So why not move it to assets?
Otto 2:13 pm on September 15, 2012 Permalink |
No, the readme.txt file belongs with the plugin file itself and should be in the ZIP file. Moving it to assets makes no sense.
The readme.txt will remain where it is.
Lance Cleveland 1:28 pm on September 17, 2012 Permalink |
Great decision. It is smart thinking like this that keeps WordPress ahead of all others and why I continue to recommend it as the best-of-breed for web publishing for my clients.
Aaron D. Campbell 2:08 pm on September 17, 2012 Permalink |
This was a great change. One of my plugins went from a 168k download to 52k. Thanks Otto.
wiredimpact 9:26 pm on September 28, 2012 Permalink |
This is a welcome change. It will greatly decrease the size of plugins. Thanks.
Aman 8:43 am on October 4, 2012 Permalink |
A very great decision
Less size more downloads
Ben 7:50 pm on October 14, 2012 Permalink |
Awesome news!
I was wondering if you guys where thinking about doing something for the languages files as well?
I know that means also adapting the way a plugin is installed as well, but the more translations the heavier your plugins gets. Just leaving the .mo files it can be about 100k per language depending on the plugin’s size. Translate your plugin in 10 languages and Bam you get 1Mo extra.
Thanks!
Otto 5:42 pm on October 16, 2012 Permalink |
No particular plans for languages, but that would be pretty cool. I was thinking more along the lines of having core support for language packs. Like, we could have a big GlotPress install somewhere to allow community translations of plugins and themes, then have an API to serve up-to-date .mo files, then serve them directly to users that request them, and let the core put them in the right place, etc.
Would be neat, but obviously that’s a big job there, with a lot of moving parts.
Ben 8:59 am on October 17, 2012 Permalink |
Sounds good, if we’re aiming to this kind of solution, that’s just perfect. But as you said it feels like a lot of changes and work involved.
Milan Dinić 6:34 pm on October 23, 2012 Permalink |
Still in works.
Milan Dinić 6:36 pm on October 23, 2012 Permalink |
How about displaying “Screenshots” tab only when there are screenshots, not always when “Screenshots” section of readme.txt has any content?
Joe Dolson 7:29 pm on October 22, 2012 Permalink |
The recent changes to the plug-in directory have been great — I noticed something today that I’m hoping would be a pretty quick fix: you can’t do a search on the words ‘accessibility’ or ‘accessible’. You can visit the tags, of course, but I don’t think it’s really the right response to send searches for these two terms to the results for ‘access’ — those are definitely two very distinct searches, and really should be giving different results.
Otto 7:40 pm on October 22, 2012 Permalink |
There’s no limitation on what you can search for. But the search engine considers similar words to be similar for the purposes of searching.
Joe Dolson 8:42 pm on October 22, 2012 Permalink |
Well, it’s definitely a problem — is there any way to tweak what the search engine considers to be similar? “access” and “accessibility” may be similar from a programming standpoint, but in terms of the search intent, they’re radically different – and I think it’s a serious limitation to the repository that you can’t easily find plug-ins that are intended to support accessibility.
Otto 8:57 pm on October 22, 2012 Permalink |
Well, did you try looking by tag instead?
http://wordpress.org/extend/plugins/tags/accessibility
Joe Dolson 10:25 pm on October 22, 2012 Permalink
Yes, as I said above. If the search can’t be modified, then it can’t be modified; but it would be valuable, since the search results are really fairly inaccurate for that term. There are over 1,000 results for ‘access’, but only a handful of them pertain to accessibility.