Create application based on Website

Issue

I searched and tried a lot to develop an application which uses the content of a Website. I just saw the StackExchange app, which looks like I want to develop my application. The difference between web and application is here:

Browser:

App:

As you can see, there are some differences between the Browser and the App. I hope somebody knows how to create an app like that, because after hours of searching I just found the solution of using a simple WebView (which is just a 1:1 like the browser) or to use Javascript in the app to remove some content (which is actually a bit buggy…).

To repeat: the point is, I want to get the content of a website (on start of the app) and to put it inside my application.

Cheers.

Solution

What you want to do is to scrape the websites in question by getting their html code and sorting it using some form of logic – I recomend xPath for this. then you can implement this data into some nice native interface.

You need however to be very aware that the data you get is not allways formated the way you want so all of your algorithems have to be very flexible.

the proccess can be cut into steps like this

1. retrive data from website (DefaultHttpClient and AsyncTask)
2. analyse and retrive relevant data (your relevant algorithm)
3. show data to user (Your interface implementation)

UPDATE

Bellow is some example code to fetch some data of a website it implements html-cleaner libary and you will need to implement this in your project.

class GetStationsClass extends AsyncTask<String, String, String> {

    @Override
    protected String doInBackground(String... params) {
        HttpClient httpclient  new DefaultHttpClient();
        httpclient.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
        httpclient.getParams().setParameter(CoreProtocolPNames.HTTP_ELEMENT_CHARSET, "iso-8859-1");
        HttpPost httppost  new HttpPost("http://ntlive.dk/rt/route?id786");
        httppost.setHeader("Accept-Charset", "iso-8859-1, unicode-1-1;q0.8");
        try {

            // Add your data
            List<NameValuePair> nameValuePairs  new ArrayList<NameValuePair>(3);
            httppost.setEntity(new UrlEncodedFormEntity(nameValuePairs, "utf-8"));

            // Execute HTTP Post Request
            HttpResponse response  httpclient.execute(httppost);
            int status  response.getStatusLine().getStatusCode();
            String data  "";
            if (status ! HttpStatus.SC_OK) {
                ByteArrayOutputStream ostream  new ByteArrayOutputStream();
                response.getEntity().writeTo(ostream);
                data  ostream.toString();
            } else {
                BufferedReader reader  new BufferedReader(new InputStreamReader(response.getEntity().getContent(),
                        "iso-8859-1"));

                String line  null;
                while ((line  reader.readLine()) ! null) {
                    data + line;
                }
                XPath xpath  XPathFactory.newInstance().newXPath();
                try {
                    Document document  readDocument(data);
                    NodeList nodes  (NodeList) xpath.evaluate("//*[@id\"container\"]/ul/li", document,
                            XPathConstants.NODESET);
                    for (int i  0; i < nodes.getLength(); i++) {
                        Node thisNode  nodes.item(i);
                        Log.v("",thisNode.getTextContent().trim);
                    }
                } catch (XPathExpressionException e) {
                    e.printStackTrace();
                }
            }

        } catch (Exception e) {
            e.printStackTrace();
        }

        return null;
    }

    @Override
    protected void onPostExecute(String result) {
        super.onPostExecute(result);
         //update user interface here
    }

}

private Document readDocument(String content) {
    Long timeStart  new Date().getTime();
    TagNode tagNode  new HtmlCleaner().clean(content);

    Document doc  null;
    try {
        doc  new DomSerializer(new CleanerProperties()).createDOM(tagNode);
        return doc;
    } catch (ParserConfigurationException e) {
        e.printStackTrace();
    } catch (Exception e) {
        e.printStackTrace();
    }

    return doc;
}

to run the code above use

new getStationsClass.execute();

Answered By – elemianus