During my PhD years I had the opportunity to intern at Microsoft Research and oDesk.
Research Scientist at Microsoft Research (Summer 2013)
In summer of 2013 I joined the Search Labs group at Microsoft Research (Mountain View Campus). There I worked with Anitha Kannan and Krishnaram Kenthapadi on algorithmically augmenting education textbooks with videos. My work has lead to two publications and one technical report (see full list of publications here).
Data Scientist at oDesk.com (Summer 2012)
In summer of 2012 I joined oDesk Research. There I worked with Panagiotis Papadimitriou on building a framework for recommending potential good contractors to employers. The framework we developed is currently in use by oDesk. Furthermore, my involvment in this project sparked my interest in hiring decisions in Online Labor Markets, a subject that ended up being abig chapter of my thesis. For related publications see here.
Data Scientist at oDesk.com (Summer 2011)
In summer of 2011 I joined for the first time oDesk Research. There I worked with John Horton on finding associations amongst the available skills in the market place. This was my first exposure to online labor markets. The very good collaboration with John, as well as the exciting questions that were naturally arising is this understudied area of paid crowdsourcing were a few of the fundamental reasons I ended up pursuing a disertation about inefficiencies in Online Labor Markets.
- Panos Ipeirotis (since 2010, Advisor)
- John Horton (since 2011, Mentor at oDesk, now Professor at NYU)
- Panagiotis Papadimitriou (since 2012, Mentor at oDesk, now working on "Hiring Decisions")
- Anitha Kannan (summer 2013, Microsoft Research)
- Krishnaram Kenthapadi (summer 2013, Microsoft Research)
- Michalis Faloutsos (2008-2010, M.Sc. advisor)
- Theodora Varvarigou (2007-2008, Undergrad advisor)
In the last four years I had the opportunity to teach and TA a series of classes related to information systems, data mining and big data:
- Dealing with Data (Spring 2014): Professor. Feedback score: 6.4/7.0
- Networks Crowds and Markets (Spring 2014): Teaching Fellow
- Networks Crowds and Markets (Spring 2013): Teaching Fellow
- Practical Data Science (Fall 2012): Teaching Fellow
- Information Technology in Business and Society (Fall 2011): Teaching Fellow
- Datamining for Business Intelligence (Spring 2011): Teaching Fellow
Update (12/17/2014): I have accepted a job at Boston College as an assistant professor.I am a fifth year PhD candidate in Information Systems at NYU Stern school of business. My advisor is Panos Ipeirotis.
My research focuses on identifying and resolving inefficiencies in online labor markets (read more). Broader areas of interest include machine learning, text mining, crowdsourcing and the economics of IT.
Here is a two-minute video about my research:
For a complete list of publications click here.
My research deals with problems that arise in online labor markets (such as oDesk). Online labor markerplaces (OLMs) allow employers to connect with freelancers around the globe to accomplish tasks that span diverse categories such as web development, writing and translation, accounting, etc. These marketplaces are growing fast and the freelancer annual earnings are expected to grow from $1 billion in 2012 to $10 billion by 2020. A typical scenario in these workplaces involves an employer posting a task, multiple contractors bidding for it, some (or one) of them getting hired, completing the task online and finally receiving a payment.The focus of my work is on on understanding and explaining the inefficiencies that appear in OLMs, and proposing applicable solutions that will develop a frictionless, mutually beneficial workplace for both the workers, the employers and the marketplace itself. A brief discription of the three questions I am currently working on is given below.
What happens when a worker switches to a new type of task, for which the worker has no prior history? Are reputations transferable across categories and predictive of future performance?Read more »
Career development paths
How can we quantify the expected utility distributions for every combination of skill and level of expertise that appear in an OLM? How can we use these utilities to recommend user-specific "development paths"?
How can employers make hiring decisions? How can we redistribute workers and employers in order to maximize both the number of hires as well as the longterm overall satisfaction of the involved parties (workers, employers and the marketplace)?
Related Publications/working papers
- Kokkodis Marios and Ipeirotis G. Panagiotis. "Reputation Transferability in Online Labor Markets". Second round (major revision) in Management Science.
- Kokkodis Marios and Ipeirotis G. Panagiotis. "How well can you do? Predicting user performance using inter-category information". Proceedings of the sixth ACM international conference on Web search and data mining. ACM, 2013.
- Kokkodis Marios and Ipeirotis, G. Panagiotis. "Where are you coming from, stranger? Predicting future performance using inter-category reputation". In 2012 Winter Conference on Business Intelligence.
In recent years, online marketplaces have experienced (and continue to experience) a significant growth in their transaction volume. As a significant new entrant, online labor marketplaces, such as oDesk, Mechanical Turk, and TaskRabbit, follow this trend as well. More precisely, statistics from oDesk, which has the largest revenue share in online workplaces, show an exponential growth in total hours worked per week since 2004; for 2012, the company was reporting transactions of more than 500,000 hours of work-time billed per week.
On a similar note, Mechanical Turk, receives hundreds of thousands of dollars worth of new jobs every day. A key difference of online labor markets with other marketplaces is that work is mainly an ``experience good'' meaning it is difficult to observe the quality of the deliverable in advance; a key solution to resolve this uncertainty is the use of reputation systems. Reputation systems provide signals about the past performance of workers. Such signals are commonly predictive of the quality of users' future performance, in a wide variety of online communities e.g., online reviews, question and answering communities and others. Consequently, it is rational to assume that employers, who have limited knowledge of the skills and abilities of a remote contractor, often consult the history of past transactions to better understand whether a contractor is qualified and suitable for the task at hand.
The implicit assumption of most existing reputation systems is that the past working history, for which a participant has been rated for, is similar to the future tasks in which the participant will engage in. However, in many online marketplaces, the tasks that are completed span across a variety of different categories: web development, writing and translation, sales and marketing and so on. Such an assortment naturally forms a highly heterogeneous workplace environment. So, what happens when, say, a worker switches to a new type of task? For example, what happens when a contractor with a background in web development decides to work on a graphic design task? What can we say regarding the possible outcome of a programming task, for a worker with history in technical writing? In general, are reputations transferable across categories and predictive of future performance? How can we estimate task affinity and use the information to best estimate expectations of future performance?
This question also applies to ``offline'' work, which increasingly leaves traces in online settings (e.g., through profiles on LinkedIn, or online resumes on Monster). As workers progress in their careers they often transition from one type of job to another (e.g., an engineer to a managerial position). Being able to understand how past performance in one type of job signals ability to perform in another, can improve significantly our ability to staff positions and allocate the right workers to right positions.
A key contribution of the paper is the presented framework that allows existing rating systems explicitly use the type of task that is associated with a past rating. We propose a set of predictive models that use Bayesian inference to estimate the future performance of a user, based on category-specific past performance. Specifically, we assume that the category-specific qualities (or skills) of a user are latent and not directly observable. However, these skills are reflected into a set of other measurable characteristics, such as employer ratings for past projects. Based on these past ratings, we build models that are capable of connecting past performance across categories to predict performance in a new category for which we have either no, or very few, past data points. We present models of increasing complexity, starting with the assumption that the latent qualities are static, but then alleviate this assumption, allowing the latent qualities to with time or gained experience. In particular, we use a linear dynamical system, which provide predictions that incorporate the dynamic behavior of latent qualities. Since for many category pairs we lack sufficient data to build robust predictors (e.g., from English-Russian translation to web development), we also build and present a hierarchical scheme that compensates for the inherent sparseness by using training data from higher-level, more general categories, to compute the cross-category predictive power of past ratings. For our experimental evaluation we use a unique dataset of real transactional oDesk data. In particular, this dataset consists over a million of real transactions across tens of different categories from the oDesk marketplace. These transactions capture histories of hundreds of thousands of different contractors. We build and evaluate our models on this data and clearly demonstrate how different categories are correlated with each other, and whether past performance in a given category contains predictive information about performance in another. We further compare our models with the existing baseline of uniformly averaging past reputation and we show that our models perform significantly better, providing up to 25\% improvement over the baseline in terms of mean absolute error. Finally, to examine the robustness of our models we run simulations by changing the input distribution. The simulation results give us further confidence regarding the adaptiveness of our models, as well as with very insightful information about the performance and appropriateness of each one of our approaches.
Our study contributes to managerial decision making in these and similar workplaces. In particular, our analysis shows a clear and methodologically sound approach for analyzing the correlations between different task categories, and as a result, we provide a more accurate estimate of a worker's performance in a new category. This information is valuable among employers that participate in online labor markets, allowing them to make safer and better informed hiring decisions. On a parallel trajectory, our analysis can be also used by these marketplaces as a guideline to reduce friction by recommending to contractors to apply for tasks that are seemingly out of their scope, but for which these contractors are highly likely to provide successful outcomes. The overall conclusion is simple: Reputation schemes stand to benefit significantly if they adjust the feedback scores of the participating users to take into account the type of task that a user is expected to complete (or has already completed), as well as the user's past category-specific performance history.
Related Publications/working papers
- Kokkodis Marios and Ipeirotis G. Panagiotis. "Career Development paths in Online Labor Markets". (working paper)
- Kokkodis Marios and Ipeirotis G. Panagiotis. "The Utility of Skills in Online Labor Markets". ICIS 2014
Online labor marketplaces (OLMs) such as oDesk.com, Elance.com, and Freelancer.com allow employers to connect with freelancers around the globe to accomplish tasks that span diverse categories such as web development, writing and translation, accounting, etc. These marketplaces are growing fast and the freelancer annual earnings are expected to grow from $1 billion in 2012 to $10 billion by 2020. A typical scenario in these workplaces involves an employer posting a task, multiple contractors bidding for it, some (or one) of them getting hired, completing the task online and finally receiving a payment.
In sync with offline workplaces, work in OLMs is an ``experience good'', meaning it is practically impossible to know the quality of the task outcome (or even the expertise of a contractor in a given skill) in advance. A key solution to resolve this uncertainty is the use of online reputation systems, which provide signals about the past performance of workers. However, reputation systems in these marketplaces fail to capture the actual worker quality: they tend to be highly skewed towards high values, and they eventually become uninformative.
The contractors' value in OLMs resides in a combination of both observable and latent characteristics. The observed characteristics usually include a list of skills, the educational background, the work history and the certifications of the applicant. The latent characteristics include the freelancer's expertise and true ability on the listed qualifications. Very similar to the offline setting, the demand and supply distributions (and as a result the expected payoff) of each contractor with a given set of skills and a given level of expertise are very heterogeneous; for example a Java expert might have a very different expected payoff than an expert in customer service support. Similarly, a c# expert might have a higher expected payoff than a c# beginner, etc.
This observation leads to two basic questions: (1) how can we estimate the latent level of expertise of a given contractor and a given skill? And (2), how can we quantify the value of a skill in an online labor marketplace and how is this value correlated with the level of expertise of a contractor?
In this work, we focus on addressing these two questions. We first propose that the utility of each skill is strongly correlated with the level of expertise of a specific worker, an assumption that also holds in offline markets. Based on the intuition that an experienced worker should on expectation receive higher compensations than a beginner, we formally define the conditional utility of a skill given a certain level of expertise. However, the actual level of expertise for a given skill and a given contractor is latent (not directly observable) and dynamic (evolves over time). To overcome this, we use a series of directly observed characteristics that intuitively are related with the level of expertise of a given skill. Based on these observable characteristics, we propose to build a Hidden Markov Model, which estimates the latent and dynamic levels of expertise of each contractor on a given skill.
For the deployment and evaluation of our methodology we use a unique transactional dataset of 1.5 million job applications from the biggest (in terms of contractor earnings) online labor market oDesk.com. We compare our proposed approach with two baselines, and show that our framework performs significantly better, and provide predictions of contractors’ levels of expertise with really high accuracies. Once we compute the level of expertise of each contractor, we estimate the conditional utility of each skill in our dataset. We finally discuss how certain skills appear to have a much higher expected compensation once someone masters them than others.
Our analysis is the first that quantifies the value of a series of skills. We firmly believe that both online labor marketplaces and contractors can benefit significantly from this analysis. We finally acknowledge that this study is the first step of answering a much bigger question: given a set of skills with a certain level of expertise, what should the next steps of a utility-optimizing contractor in (but also beyond) an online labor marketplace be?
The next steps include the development of a skills' recommendation framework. In particular, consider a scenario where at any given time, workers have two options: (1) to exploit their current skillset and expertise by getting hired and completing a task, or (2), invest their time on improving/expanding their skillset and expect future increased returns. What is the optimal decision for each worker? Exploit or improve? We plan to develop a framework that captures this behavior and recommends the optimal decision for the contractor. Such a system could be used as a career development adviser.
Related Publications/working papers
- Kokkodis Marios, Papadimitriou Panagiotis and Ipeirotis G. Panagiotis. "On hiring decisions in Online Labor Markets". (working paper)
- Kokkodis Marios, Papadimitriou Panagiotis, Antonellis Ioannis, Ipeirtotis G. Panagiotis. "Who Should I Hire? Maximizing Employer's Utility in Online Labor Markets". In 2013 Winter Conference on Business Intelligence.
Online labor marketplaces (OLMs) such as oDesk.com and freelancer.com, allow employers to connect with workers around the globe to accomplish diverse tasks including web development, writing and translation, accounting, etc. These marketplaces are growing fast and the worker annual earnings are expected to grow from $1 billion in 2012 to $10 billion by 2020.
Although such platforms have provided the employers with a solution to the scarcity of local talent, they have not really changed the process that employers have to go through to source the ideal candidates for their tasks: an employer needs initially to describe the job requirements and create an opening, to which workers that are looking for opportunities can apply. Then, the employer has to (1) review all applicants by looking at their online profile information and/or by personally interviewing them, and (2) come up with a hiring decision.
Similar to the offline workplaces, to evaluate an applicant, the employer has to assess both observed and latent characteristics. The observed characteristics usually include a list of skills, the educational background, the work history and the certifications or tests that the applicant has successfully passed. The latent characteristics include the worker's quality and true ability on his listed qualifications. The existence of latent characteristics, the heterogeneity that appears in the observed ones as well as the interactions between the two make the matching process a very challenging task; Hiring decisions are based on manually shaped expectations of complicated similarities between job openings, employers, and workers. These expectations usually come with high uncertainties, since performing a task is an ``experience good'': for both the worker and the employer, it is practically infeasible to know the outcome of their collaboration in advance.
To minimize the level of uncertainty, most of the online labor marketplaces have developed reputation systems. Workers get rated for the tasks they accomplish and these ratings become part of their online resumes. Employers can then get a better picture of the workers' past performance and make better-informed hiring decisions. However, in online labor markets, as well as in most of the online markets in general, reputation scores are very skewed towards high ratings (J-shape distributions), and as a result they become almost uninformative. Since reputation systems fail to provide insightful information about the workers' quality, how do employers make hiring decisions? What are the characteristics that employers value the most?
In this work we focus on answering these questions. We propose a series of increasing complexity predictive models that describe employers' hiring decisions. We start our analysis by assuming that employers are rational utility maximizers; Their utility is straightforwardly maximized along with the probability of selecting the best possible applicant for each specific opening. Based on this assumption, we first propose a ranking aggregator that ranks candidates in all the available dimensions and then aggregates these ranks to create a global ranking. Next, we draw on empirical economics and propose a Logit model and finally, we built a probabilistic graphical model (Bayesian network). We compare our models with the vanilla reputation score baseline, where each employer ranks the available applicants based on their previously collected feedback score.
We train and test all proposed approaches on a unique dataset of real transactional oDesk data. In particular, this dataset consists of roughly 1.5 million job applications on more than 90,000 openings related to four different task categories. Our dataset includes both openings that lead to a single hire, but also, openings where the employer chose to hire multiple workers. We use different evaluation metrics and show evidence that our models significantly outperform the vanilla reputation baseline. We further perform an econometric analysis, as well as a propensity score matching study, and observe that the attributes that have the strongest positive effect on hiring probability are whether or not the worker and the employer have previously worked together, the available information on the worker's profile, the countries of the employer and the worker and the skillset of the worker. Finally, our analysis shows that the faster the worker applies to an opening, the higher is the probability to get hired.
Our work is the first to study the effect of different characteristics on hiring decisions by incorporating a massive amount of observational data. We believe that understanding how hiring decisions are made is beneficial for both the employers and the workers, and critical for the marketplace. In particular, by developing approaches that estimate the applicants' hiring probabilities: (1) employers will be able to make better-informed and faster decisions based on the suggested applicants' rankings, (2) workers will save time by not applying to openings that have very low hiring probability and (3) the marketplace might identify weaknesses in workers' profiles (e.g., skills not reported, the profile description is not sufficient etc.) and suggest targeted profile improvements based on how each profile characteristic affect hiring decisions. As a result, our proposed approaches will minimize the friction in the marketplace and increase both the marketplace's transaction volume as well as the overall satisfaction of the workers and the employers.
Journal Publications (working/submitted)
- Kokkodis Marios and Ipeirotis Panos. "Reputation Transferability in Online Labor Markets". Third round (minor revision) in Management Science.
- Kokkodis Marios, Papadimitriou Panagiotis and Ipeirotis Panos. "On hiring decisions in Online Labor Markets". (submitted at a top IS journal)
- Kokkodis Marios and Ipeirotis Panos. "Career Development paths in Online Labor Markets". (working paper, draft available)
Conference Publications with Proceedings
- Kokkodis Marios, Papadimitriou Panagiotis and Ipeirotis G. Panagiotis. “Hiring Behavior Models for Online Labor Markets”. Proceedings of the eighth ACM international conference on Web search and data mining (WSDM). ACM, 2015
- Kokkodis Marios and Ipeirotis G. Panagiotis. "The Utility of Skills in Online Labor Markets". Proceedings of the International Conference on Information Systems (ICIS), 2014
- Kokkodis Marios, Kannan Anitha, and Krishnaram Kenthapadi. "Assigning Educational Videos at Appropriate Locations in Textbooks". Proceedings of the Educational Data Minining conference. 2014
- Kokkodis Marios, Kannan Anitha, and Krishnaram Kenthapadi. "Assigning Videos to Textbooks at Appropriate Granularity". Proceedings of the first ACM Learning at Scale conference. ACM, 2014
- Kokkodis Marios and Ipeirotis Panos. "How well can you do? Predicting user performance using inter-category information". Proceedings of the sixth ACM international conference on Web search and data mining. ACM, 2013.
- Kokkodis Marios. "Learning from positive and unlabeled amazon reviews: towards identifying trustworthy reviewers". Proceedings of the 21st international conference companion on World Wide Web. ACM, 2012.
Refereed Conferences without Proceedings
- Kokkodis Marios, Papadimitriou Panagiotis, Ipeirotis, G, Panagiotis. “On Hiring Decisions in Online Labor Markets”. Workshop on Information Systems and Economics (WISE), 2014.
- Kokkodis Marios. “Online Labor Markets: Reputation Transferability, Career Development Paths and Hiring Decisions”. Doctoral Consortium, AAAI Conference on Human Computation & Crowdsourcing (HCOMP), 2014.
- Kokkodis Marios, Papadimitriou Panagiotis, Ipeirotis, G, Panagiotis. “Hiring Behavior Models for Online Labor Markets”. Winter Conference on Business Intelligence, 2014.
- Kokkodis Marios, Papadimitriou Panagiotis, Antonellis Ioannis, Ipeirtotis Panos. "Who Should I Hire? Maximizing Employer's Utility in Online Labor Markets". Winter Conference on Business Intelligence, 2013.
- Kokkodis, Marios. "Credibility dimensions in online reviews". Winter Conference on Business Intelligence, 2013.
- Kokkodis Marios and Ipeirotis Panos. "Where are you coming from, stranger? Predicting future performance using inter-category reputation". Winter Conference on Business Intelligence, 2012.
- Marios Kokkodis, Anitha Kannan, and Krishnaram Kenthapadi, Assigning Educational Videos at Appropriate Locations in Textbooks, no. MSR-TR-2014-62, July 2014
- "Maximizing Employer's Utility in Online Labor Markets". INFORMS 2013
- "Reputation Transferability in Online Labor Markets". INFORMS 2013
M.Sc. Thesis Publications
- Kokkodis Marios, Faloutsos Michalis and Markopoulou Athina."Network-level characteristics of Spamming: An empirical analysis", IEEE Network Protocols (ICNP), Vancouver, BC, October 2011.
- Kokkodis Marios and Faloutsos Michalis."Spamming Botnets: Are we losing the war?", CEAS, Mountain View, CA, July 2009.
- Kokkodis Marios and Faloutsos Michalis "A latest update on Spamming Botnets." Virus Bulletin magazine, October 2009.